[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:02 dist-test-slave-28bv krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:02 dist-test-slave-28bv krb5kdc[399](info): set up 2 sockets
Jun 20 00:12:02 dist-test-slave-28bv krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 00:12:03.122784 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:03.256907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:03.341030 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:03.341274 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:03.341298 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:03.342543 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:03.342595 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:03.342610 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:03.342622 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:03.342635 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:03.344205 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:03.344512 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:03.344803 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:03.351897 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.352066 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.352098 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.352342 412 server_base.cc:1034] running on GCE node
I20260620 00:12:03.352741 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:03.353085 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:03.354266 412 hybrid_clock.cc:648] HybridClock initialized: now 1781914323354261 us; error 36 us; skew 500 ppm
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:03.358072 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:03.358692 412 webserver.cc:460] Webserver started at http://127.0.98.62:36729/ using document root <none> and password file <none>
I20260620 00:12:03.358911 412 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:03.358958 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:03.359058 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:03.360209 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "f3379786b45b4b6695da7acfae2cb03a"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "dfa1ed095036d67825b7bb9831b049e3"
server_key_iv: "1705f536a6fa9b6953c85699958bda13"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.360680 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3379786b45b4b6695da7acfae2cb03a"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "dfa1ed095036d67825b7bb9831b049e3"
server_key_iv: "1705f536a6fa9b6953c85699958bda13"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.362176 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:03.362967 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.363168 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:03.363247 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "f3379786b45b4b6695da7acfae2cb03a"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "dfa1ed095036d67825b7bb9831b049e3"
server_key_iv: "1705f536a6fa9b6953c85699958bda13"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.363310 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:03.375000 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:03.377863 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:03.378067 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:03.384402 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38411
I20260620 00:12:03.384513 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38411 every 8 connection(s)
I20260620 00:12:03.384904 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:03.385908 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:03.388289 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a: Bootstrap starting.
I20260620 00:12:03.388945 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:03.389185 480 log.cc:824] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:03.389966 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a: No bootstrap required, opened a new log
I20260620 00:12:03.391624 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260620 00:12:03.391779 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:03.391803 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3379786b45b4b6695da7acfae2cb03a, State: Initialized, Role: FOLLOWER
I20260620 00:12:03.391916 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260620 00:12:03.391968 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:03.391994 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:03.392026 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:03.392520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 412
I20260620 00:12:03.392638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:03.392675 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260620 00:12:03.392798 480 leader_election.cc:302] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3379786b45b4b6695da7acfae2cb03a; no voters:
I20260620 00:12:03.392978 480 leader_election.cc:288] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:03.393020 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:03.393136 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [term 1 LEADER]: Becoming Leader. State: Replica: f3379786b45b4b6695da7acfae2cb03a, State: Running, Role: LEADER
I20260620 00:12:03.393220 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260620 00:12:03.393265 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:03.393536 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3379786b45b4b6695da7acfae2cb03a. Latest consensus state: current_term: 1 leader_uuid: "f3379786b45b4b6695da7acfae2cb03a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } } }
I20260620 00:12:03.393563 392 external_mini_cluster.cc:1428] Setting key f58bc7237a1cfc520f9d91b21b9a63c9
I20260620 00:12:03.393610 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:03.393829 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3379786b45b4b6695da7acfae2cb03a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3379786b45b4b6695da7acfae2cb03a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } } }
I20260620 00:12:03.393985 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:03.394747 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:03.395403 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:03.396870 487 catalog_manager.cc:1347] Generated new cluster ID: e23203b3037a453ab91c594aaff0f0e1
I20260620 00:12:03.396926 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:03.412755 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:03.412763 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.394527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.62:38411)
0620 00:12:03.395006 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.395016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.395053 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:12:03.397331 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.397339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.397396 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.397541 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.399669 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.401176 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.402763 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.403386 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.403487 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.408241 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.408245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.408246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.408270 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.410365 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.410967 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.410970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.410971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.411084 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.411377 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.411378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.411378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.411691 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.411823 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.412169 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.412325 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":65,"threads_started":1}
I20260620 00:12:03.413498 487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:03.432516 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3379786b45b4b6695da7acfae2cb03a: Generated new TSK 0
I20260620 00:12:03.432881 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:03.472911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:03.561455 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:03.561710 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:03.561761 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:03.563015 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:03.563087 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:03.563135 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:03.564733 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:03.565022 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:03.565269 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:03.571084 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.570984 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.570974 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.571259 506 server_base.cc:1034] running on GCE node
I20260620 00:12:03.571511 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:03.571770 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:03.572918 506 hybrid_clock.cc:648] HybridClock initialized: now 1781914323572898 us; error 37 us; skew 500 ppm
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:03.575615 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:03.576171 506 webserver.cc:460] Webserver started at http://127.0.98.1:46631/ using document root <none> and password file <none>
I20260620 00:12:03.576438 506 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:03.576509 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:03.576713 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:03.577791 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "38d39e425f734ac0a4546fc0143b0958"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "7a94de67dea1d1aaa0bec716f76e952a"
server_key_iv: "7c86cadbec68375126f60cca88bc62a1"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.578171 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38d39e425f734ac0a4546fc0143b0958"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "7a94de67dea1d1aaa0bec716f76e952a"
server_key_iv: "7c86cadbec68375126f60cca88bc62a1"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.579536 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:03.580262 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.580442 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:03.580541 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "38d39e425f734ac0a4546fc0143b0958"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "7a94de67dea1d1aaa0bec716f76e952a"
server_key_iv: "7c86cadbec68375126f60cca88bc62a1"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.580654 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:03.590626 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:03.593150 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:03.593360 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:03.593618 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:03.593956 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:03.594012 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.594055 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:03.594085 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.600293 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35865
I20260620 00:12:03.600344 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35865 every 8 connection(s)
I20260620 00:12:03.600762 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:03.608418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 506
I20260620 00:12:03.608522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:03.608723 392 external_mini_cluster.cc:1428] Setting key 50bef44df48bfb808a94ed3cdd44bf00
I20260620 00:12:03.611132 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.601586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41001 (local address 127.0.98.62:38411)
0620 00:12:03.601661 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.601663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.601872 (+ 209us) server_negotiation.cc:407] Connection header received
0620 00:12:03.602450 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.602452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.602469 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.602543 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.603586 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.604100 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.605249 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.605462 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.605538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.607808 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.607810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.607810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.607818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.609596 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.610051 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.610052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.610053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.610118 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.610392 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.610393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.610393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.610553 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.610621 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.610940 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.611038 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:12:03.611387 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.601582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.1:41001)
0620 00:12:03.601740 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:03.601786 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:03.602306 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:03.602626 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:03.602630 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:03.603255 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:03.603464 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.603470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.604203 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.604203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.605136 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.605139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.605558 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.605559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.605691 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.606208 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:03.606217 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:03.607692 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:03.609693 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.609700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.609702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.609969 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.610196 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.610196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.610196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.610328 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.610615 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:03.610619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:03.610788 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 00:12:03.610911 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:12:03.611082 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 00:12:03.611960 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260620 00:12:03.612078 635 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:03.612294 635 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
I20260620 00:12:03.612946 444 ts_manager.cc:194] Registered new tserver with Master: 38d39e425f734ac0a4546fc0143b0958 (127.0.98.1:35865)
I20260620 00:12:03.613571 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41001
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:03.636809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:03.721815 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:03.722043 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:03.722071 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:03.723287 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:03.723335 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:03.723363 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:03.724885 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:03.725092 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:03.725287 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:03.730562 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.730628 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.730736 640 server_base.cc:1034] running on GCE node
W20260620 00:12:03.730584 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.731108 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:03.731406 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:03.732610 640 hybrid_clock.cc:648] HybridClock initialized: now 1781914323732588 us; error 36 us; skew 500 ppm
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:03.735314 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:03.735785 640 webserver.cc:460] Webserver started at http://127.0.98.2:46071/ using document root <none> and password file <none>
I20260620 00:12:03.736022 640 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:03.736069 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:03.736210 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:03.737378 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "0db204b8f59b4e79b275b3eba477fb18"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "18e0978c1f116b848a3df25dae74ecc9"
server_key_iv: "a89f7db460ba6dc8c5ee8174952b2fba"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.737756 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0db204b8f59b4e79b275b3eba477fb18"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "18e0978c1f116b848a3df25dae74ecc9"
server_key_iv: "a89f7db460ba6dc8c5ee8174952b2fba"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.739192 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:03.740060 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.740245 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:03.740366 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "0db204b8f59b4e79b275b3eba477fb18"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "18e0978c1f116b848a3df25dae74ecc9"
server_key_iv: "a89f7db460ba6dc8c5ee8174952b2fba"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.740449 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:03.749397 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:03.751796 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:03.751969 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:03.752216 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:03.752578 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:03.752640 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.752696 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:03.752744 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.759387 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36667
I20260620 00:12:03.759472 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36667 every 8 connection(s)
I20260620 00:12:03.759831 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:03.762064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 640
I20260620 00:12:03.762149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:03.762374 392 external_mini_cluster.cc:1428] Setting key 32cabda6353b41aea017d877845ec6e3
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:03.771181 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.760762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34411 (local address 127.0.98.62:38411)
0620 00:12:03.760838 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.760840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.761053 (+ 213us) server_negotiation.cc:407] Connection header received
0620 00:12:03.761553 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.761554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.761564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.761625 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.762711 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.763229 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.764308 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.764657 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.764741 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.767513 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.767515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.767515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.767524 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.769492 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.770014 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.770016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.770016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.770085 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.770396 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.770397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.770397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.770568 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.770647 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.770978 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.771077 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:12:03.771324 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.760768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.2:34411)
0620 00:12:03.760918 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:03.760956 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:03.761438 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:03.761684 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:03.761688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:03.762396 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:03.762639 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.762644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.763323 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.763324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.764212 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.764215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.764793 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.764794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.764996 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.765582 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:03.765595 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:03.767397 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:03.769634 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.769642 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.769645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.769924 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.770182 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.770182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.770183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.770316 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.770670 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:03.770677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:03.770828 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:12:03.770933 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 00:12:03.771075 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260620 00:12:03.771915 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260620 00:12:03.772012 769 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:03.772214 769 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:12:03.772717 444 ts_manager.cc:194] Registered new tserver with Master: 0db204b8f59b4e79b275b3eba477fb18 (127.0.98.2:36667)
I20260620 00:12:03.773072 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:03.792515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:03.877853 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:03.878094 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:03.878122 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:03.879317 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:03.879359 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:03.879388 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:03.880970 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:03.881179 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:03.881379 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:03.886721 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.886839 774 server_base.cc:1034] running on GCE node
W20260620 00:12:03.886775 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:03.886952 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:03.887199 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:03.887488 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:03.888650 774 hybrid_clock.cc:648] HybridClock initialized: now 1781914323888635 us; error 36 us; skew 500 ppm
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:03.891618 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:03.892155 774 webserver.cc:460] Webserver started at http://127.0.98.3:42595/ using document root <none> and password file <none>
I20260620 00:12:03.892350 774 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:03.892391 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:03.892488 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:03.893756 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "0aad3b5795094ed08dc1a9645ac4d364"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "be320f5890888d37b7baad1b8d07ad9a"
server_key_iv: "1ac7bb1dc1a6928547b4eaf0c3fc503c"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.894184 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0aad3b5795094ed08dc1a9645ac4d364"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "be320f5890888d37b7baad1b8d07ad9a"
server_key_iv: "1ac7bb1dc1a6928547b4eaf0c3fc503c"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.895743 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:03.896613 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.896849 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:03.896944 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "0aad3b5795094ed08dc1a9645ac4d364"
format_stamp: "Formatted at 2026-06-20 00:12:03 on dist-test-slave-28bv"
server_key: "be320f5890888d37b7baad1b8d07ad9a"
server_key_iv: "1ac7bb1dc1a6928547b4eaf0c3fc503c"
server_key_version: "encryptionkey@0"
I20260620 00:12:03.896996 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:03.907721 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:03.909929 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:03.910049 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:03.910290 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:03.910559 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:03.910591 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.910614 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:03.910626 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:03.917419 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36703
I20260620 00:12:03.917500 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36703 every 8 connection(s)
I20260620 00:12:03.917819 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:03.927826 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.918733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39653 (local address 127.0.98.62:38411)
0620 00:12:03.918845 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.918848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.919032 (+ 184us) server_negotiation.cc:407] Connection header received
0620 00:12:03.919553 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.919554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.919566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.919633 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.920664 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.921174 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.922265 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.922460 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.922520 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.924885 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.924886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.924886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.924894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.926547 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.926989 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.926990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.926991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.927046 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.927267 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.927268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.927268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.927382 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.927447 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.927691 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.927757 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 00:12:03.928118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 774
I20260620 00:12:03.928158 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.918699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.3:39653)
0620 00:12:03.918903 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:03.918946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:03.919440 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:03.919711 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:03.919714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:03.920316 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:03.920526 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.920531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.921291 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.921292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.922138 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:03.922141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.922539 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.922540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:03.922646 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.923260 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:03.923268 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:03.924764 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:03.926666 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.926673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.926675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.926913 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.927109 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:03.927109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:03.927110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:03.927212 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:03.927445 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:03.927449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:03.927576 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 00:12:03.927661 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 00:12:03.927797 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":136,"threads_started":1}
I20260620 00:12:03.928207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:03.928385 392 external_mini_cluster.cc:1428] Setting key 94182572baa2a71d9d908731a72d87b0
I20260620 00:12:03.928802 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260620 00:12:03.928928 903 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:03.929176 903 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
I20260620 00:12:03.929554 444 ts_manager.cc:194] Registered new tserver with Master: 0aad3b5795094ed08dc1a9645ac4d364 (127.0.98.3:36703)
I20260620 00:12:03.929960 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39653
I20260620 00:12:03.940770 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:03.965595 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.957086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.98.1:35865)
0620 00:12:03.957297 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.957302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.957314 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:03.957369 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.957371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.957385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.957455 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.958029 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.958769 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.959617 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.960002 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.960070 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.962589 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.962599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.962599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.962607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.964313 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.964745 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.964749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.964749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.964845 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.965059 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.965060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.965060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.965202 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.965329 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.965442 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.965510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
W20260620 00:12:03.966056 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39534
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:12:03 dist-test-slave-28bv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:03.990453 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:03.982575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.98.1:35865)
0620 00:12:03.982641 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:03.982647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:03.982694 (+ 47us) server_negotiation.cc:407] Connection header received
0620 00:12:03.982724 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:03.982725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:03.982737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:03.982784 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:03.983484 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.984059 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.984977 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:03.985182 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:03.985275 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:03.987789 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:03.987790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:03.987791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:03.987799 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:03.989328 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.989704 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.989705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.989706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.989771 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:03.990015 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:03.990016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:03.990016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:03.990130 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:03.990240 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:03.990314 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 00:12:03.990384 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:12:03.991104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 506
I20260620 00:12:03.996393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 640
I20260620 00:12:04.002596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 774
I20260620 00:12:04.007375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 412
2026-06-20T00:12:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1962 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:04 dist-test-slave-28bv krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:04 dist-test-slave-28bv krb5kdc[917](info): set up 2 sockets
Jun 20 00:12:04 dist-test-slave-28bv krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 00:12:05.112105 392 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:05.239584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:05.321822 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:05.322098 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:05.322145 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:05.323367 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:05.323446 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:05.323469 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:05.323493 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:05.323521 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:05.325062 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:05.325356 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:05.325693 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:05.331094 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.331043 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.331043 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:05.331324 930 server_base.cc:1034] running on GCE node
I20260620 00:12:05.331590 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:05.331959 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:05.333149 930 hybrid_clock.cc:648] HybridClock initialized: now 1781914325333130 us; error 29 us; skew 500 ppm
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:05.336017 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:05.336627 930 webserver.cc:460] Webserver started at http://127.0.98.62:37217/ using document root <none> and password file <none>
I20260620 00:12:05.336889 930 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:05.336938 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:05.337073 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:05.338174 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "4528fc3ace104594baf83ec2679c0e2b"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "817b8ccc12cfe809669fee821df0f735"
server_key_iv: "aa00eadc4d05481c79df8ac1451a0889"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.338577 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "4528fc3ace104594baf83ec2679c0e2b"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "817b8ccc12cfe809669fee821df0f735"
server_key_iv: "aa00eadc4d05481c79df8ac1451a0889"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.340015 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:05.340965 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.341238 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:05.341353 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "4528fc3ace104594baf83ec2679c0e2b"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "817b8ccc12cfe809669fee821df0f735"
server_key_iv: "aa00eadc4d05481c79df8ac1451a0889"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.341439 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:05.347983 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:05.350351 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:05.350507 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:05.354597 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39035
I20260620 00:12:05.354689 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39035 every 8 connection(s)
I20260620 00:12:05.354991 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:05.355875 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:05.358284 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: Bootstrap starting.
I20260620 00:12:05.358888 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:05.359239 998 log.cc:824] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:05.359998 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: No bootstrap required, opened a new log
I20260620 00:12:05.361618 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260620 00:12:05.361800 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:05.361853 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4528fc3ace104594baf83ec2679c0e2b, State: Initialized, Role: FOLLOWER
I20260620 00:12:05.362033 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260620 00:12:05.362135 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:05.362180 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:05.362221 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:05.362964 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260620 00:12:05.363080 998 leader_election.cc:302] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4528fc3ace104594baf83ec2679c0e2b; no voters:
I20260620 00:12:05.363255 998 leader_election.cc:288] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:05.363428 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:05.363528 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:05.363783 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [term 1 LEADER]: Becoming Leader. State: Replica: 4528fc3ace104594baf83ec2679c0e2b, State: Running, Role: LEADER
I20260620 00:12:05.363862 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } }
I20260620 00:12:05.364209 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4528fc3ace104594baf83ec2679c0e2b. Latest consensus state: current_term: 1 leader_uuid: "4528fc3ace104594baf83ec2679c0e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } } }
I20260620 00:12:05.364281 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:05.364235 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4528fc3ace104594baf83ec2679c0e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4528fc3ace104594baf83ec2679c0e2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39035 } } }
I20260620 00:12:05.364346 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:05.364668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 930
I20260620 00:12:05.364764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:05.364953 392 external_mini_cluster.cc:1428] Setting key ab51a6e638e5c2234cb5c4a837dadd1f
W20260620 00:12:05.365792 1016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:05.365881 1016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:05.365970 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:05.366173 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:05.369820 1010 catalog_manager.cc:1347] Generated new cluster ID: d57113df923245619dff99ccf3b05b6e
I20260620 00:12:05.369884 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:05.376114 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:05.376830 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:05.378461 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.365677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.0.98.62:39035)
0620 00:12:05.368723 (+ 3046us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.368731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.368767 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:12:05.368879 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.368886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.368938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.369164 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:05.369990 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.370948 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.371739 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.372201 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.372328 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.374769 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:05.374773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:05.374774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:05.374794 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:05.376798 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.377208 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.377211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.377212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.377308 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.377525 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.377526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.377526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.377757 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:05.377946 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.378021 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.378286 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2993,"thread_start_us":67,"threads_started":1}
I20260620 00:12:05.386519 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4528fc3ace104594baf83ec2679c0e2b: Generated new TSK 0
I20260620 00:12:05.386847 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:05.421011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:05.531201 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:05.531423 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:05.531450 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:05.532780 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:05.532829 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:05.532859 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:05.534368 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:05.534632 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:05.534891 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:05.540572 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.540663 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.540581 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:05.540880 1024 server_base.cc:1034] running on GCE node
I20260620 00:12:05.541111 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:05.541370 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:05.542541 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781914325542541 us; error 42 us; skew 500 ppm
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:05.545414 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:05.546010 1024 webserver.cc:460] Webserver started at http://127.0.98.1:41439/ using document root <none> and password file <none>
I20260620 00:12:05.546258 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:05.546305 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:05.546439 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:05.547607 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "d22b0cd49b474646b30f7e47bc0d2c84"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "1e51f91a794a206d48eea74d2af99911"
server_key_iv: "c042842af998f7724eeee7b6153b100b"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.547991 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d22b0cd49b474646b30f7e47bc0d2c84"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "1e51f91a794a206d48eea74d2af99911"
server_key_iv: "c042842af998f7724eeee7b6153b100b"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.549547 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:05.550489 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.550868 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:05.550951 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "d22b0cd49b474646b30f7e47bc0d2c84"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "1e51f91a794a206d48eea74d2af99911"
server_key_iv: "c042842af998f7724eeee7b6153b100b"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.551052 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:05.561924 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:05.564309 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:05.564508 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:05.564888 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:05.565232 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:05.565289 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.565331 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:05.565358 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.571380 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37863
I20260620 00:12:05.571452 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37863 every 8 connection(s)
I20260620 00:12:05.571805 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:05.577443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1024
I20260620 00:12:05.577554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:05.577754 392 external_mini_cluster.cc:1428] Setting key 347bd33053600a4762c48d6700d3b33b
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:05.582208 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.572610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59885 (local address 127.0.98.62:39035)
0620 00:12:05.572667 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.572669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.572883 (+ 214us) server_negotiation.cc:407] Connection header received
0620 00:12:05.573370 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.573371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.573382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.573450 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:05.574359 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.574862 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.575963 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.576171 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.576226 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.578975 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:05.578977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:05.578977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:05.578986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:05.580527 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.581087 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.581088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.581088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.581142 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.581437 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.581438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.581439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.581607 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:05.581738 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.581964 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.582071 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:12:05.582279 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.572579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39035 (local address 127.0.98.1:59885)
0620 00:12:05.572778 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.572816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.573285 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.573506 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.573510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.574076 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:05.574272 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.574276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.574962 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.574963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.575868 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.575870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.576245 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.576246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.576348 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.576907 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:05.576915 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:05.578793 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:05.580644 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.580653 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.580656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.581000 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.581211 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.581212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.581213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.581360 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.581690 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:05.581695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:05.581827 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.581920 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.582055 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":126,"threads_started":1}
I20260620 00:12:05.582801 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39035
I20260620 00:12:05.582917 1153 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:05.583110 1153 heartbeater.cc:511] Master 127.0.98.62:39035 requested a full tablet report, sending...
I20260620 00:12:05.583750 962 ts_manager.cc:194] Registered new tserver with Master: d22b0cd49b474646b30f7e47bc0d2c84 (127.0.98.1:37863)
I20260620 00:12:05.584793 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:05.608810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:05.690574 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:05.690783 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:05.690806 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:05.692013 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:05.692059 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:05.692088 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:05.693599 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:05.693848 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:05.694095 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:05.698978 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.698974 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.699050 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:05.699100 1158 server_base.cc:1034] running on GCE node
I20260620 00:12:05.699399 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:05.699604 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:05.700800 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781914325700791 us; error 42 us; skew 500 ppm
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:05.703504 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:05.704075 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42213/ using document root <none> and password file <none>
I20260620 00:12:05.704286 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:05.704330 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:05.704427 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:05.706048 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "83f376a338604924b440534a8dc062f8"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "4ed287df8f346bfc1cc908336cb81306"
server_key_iv: "88c370e294dec1023dc9a856592accdf"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.706400 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83f376a338604924b440534a8dc062f8"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "4ed287df8f346bfc1cc908336cb81306"
server_key_iv: "88c370e294dec1023dc9a856592accdf"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.707826 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:05.708778 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.708971 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:05.709127 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "83f376a338604924b440534a8dc062f8"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "4ed287df8f346bfc1cc908336cb81306"
server_key_iv: "88c370e294dec1023dc9a856592accdf"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.709220 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:05.763067 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:05.765867 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:05.766045 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:05.766302 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:05.766637 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:05.766697 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.766736 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:05.766783 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.772775 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43765
I20260620 00:12:05.772866 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43765 every 8 connection(s)
I20260620 00:12:05.773216 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:05.774669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1158
I20260620 00:12:05.774773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:05.774930 392 external_mini_cluster.cc:1428] Setting key 64f8adf5a51e41d636e322194692392c
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:12:05.784307 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.774091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44389 (local address 127.0.98.62:39035)
0620 00:12:05.774163 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.774165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.774394 (+ 229us) server_negotiation.cc:407] Connection header received
0620 00:12:05.775230 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.775231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.775243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.775301 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:05.776444 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.777227 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.778599 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.778794 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.778886 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.781151 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:05.781152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:05.781153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:05.781161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:05.782708 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.783226 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.783228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.783228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.783288 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.783546 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.783547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.783548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.783678 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:05.783752 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.784100 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.784194 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:12:05.784513 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.774070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39035 (local address 127.0.98.2:44389)
0620 00:12:05.774261 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.774305 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.775011 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.775412 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.775417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.776128 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:05.776330 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.776335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.777330 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.777331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.778440 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.778443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.778894 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.778895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.779042 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.779601 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:05.779610 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:05.781051 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:05.782826 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.782833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.782835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.783127 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.783362 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.783363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.783364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.783472 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.783755 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:05.783762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:05.783907 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.784078 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.784265 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
I20260620 00:12:05.785049 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39035
I20260620 00:12:05.785175 1287 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:05.785373 1287 heartbeater.cc:511] Master 127.0.98.62:39035 requested a full tablet report, sending...
I20260620 00:12:05.785805 962 ts_manager.cc:194] Registered new tserver with Master: 83f376a338604924b440534a8dc062f8 (127.0.98.2:43765)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:12:05.786327 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:05.805028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:05.888439 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:05.888765 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:05.888793 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:05.890058 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:05.890106 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:05.890134 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:05.891541 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39035
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:05.891746 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:05.891927 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:05.897320 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:05.897320 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:05.897486 1292 server_base.cc:1034] running on GCE node
W20260620 00:12:05.897351 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:05.897780 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:05.898028 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:05.899232 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781914325899216 us; error 41 us; skew 500 ppm
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:05.901901 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:05.902357 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45699/ using document root <none> and password file <none>
I20260620 00:12:05.902544 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:05.902583 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:05.902681 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:05.903710 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ff884c536a74ca6962ce451f2776660"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "ab532b33d9a7c017139ab2798c21042f"
server_key_iv: "bb12a014bdb622c2033769735959cc0c"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.904011 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ff884c536a74ca6962ce451f2776660"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "ab532b33d9a7c017139ab2798c21042f"
server_key_iv: "bb12a014bdb622c2033769735959cc0c"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.905388 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:05.906167 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.906384 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:05.906479 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "5ff884c536a74ca6962ce451f2776660"
format_stamp: "Formatted at 2026-06-20 00:12:05 on dist-test-slave-28bv"
server_key: "ab532b33d9a7c017139ab2798c21042f"
server_key_iv: "bb12a014bdb622c2033769735959cc0c"
server_key_version: "encryptionkey@0"
I20260620 00:12:05.906558 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:05.918946 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:05.921118 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:05.921267 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:05.921525 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:05.921830 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:05.921864 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.921885 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:05.921898 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:05.927822 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39611
I20260620 00:12:05.927904 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39611 every 8 connection(s)
I20260620 00:12:05.928261 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:05.930063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1292
I20260620 00:12:05.930154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:05.930315 392 external_mini_cluster.cc:1428] Setting key 81790119f38dea3d39b09853a60b2e05
Jun 20 00:12:05 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:05.938294 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.929150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58605 (local address 127.0.98.62:39035)
0620 00:12:05.929232 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.929235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.929411 (+ 176us) server_negotiation.cc:407] Connection header received
0620 00:12:05.929870 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.929872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.929901 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.929978 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:05.931242 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.931753 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.932801 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.932999 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.933070 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.935223 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:05.935224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:05.935224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:05.935231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:05.936786 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.937272 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.937273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.937274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.937337 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.937643 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.937643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.937644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.937765 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:05.937837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.938154 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.938226 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:12:05.938446 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.929079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39035 (local address 127.0.98.3:58605)
0620 00:12:05.929279 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.929328 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.929787 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.930032 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.930035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.930895 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:05.931130 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.931155 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.931858 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.931859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.932719 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.932721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.933097 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.933098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.933210 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.933676 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:05.933684 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:05.935136 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:05.936879 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.936884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.936886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.937112 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.937447 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:05.937448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:05.937448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:05.937575 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:05.937854 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:05.937860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:05.937992 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.938102 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.938237 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260620 00:12:05.938939 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39035
I20260620 00:12:05.939055 1421 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:05.939256 1421 heartbeater.cc:511] Master 127.0.98.62:39035 requested a full tablet report, sending...
I20260620 00:12:05.939677 962 ts_manager.cc:194] Registered new tserver with Master: 5ff884c536a74ca6962ce451f2776660 (127.0.98.3:39611)
I20260620 00:12:05.940085 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58605
I20260620 00:12:05.942113 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:12:05.949538 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.943531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.98.62:39035)
0620 00:12:05.943570 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.943572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.943643 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:12:05.943704 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.943705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.943713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.943762 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:05.944300 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.944888 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.945564 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.945770 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.945828 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.946922 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:05.946923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:05.946923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:05.946930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:05.948432 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.948889 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.948890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.948890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.948959 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:05.949122 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:05.949123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:05.949123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:05.949258 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:05.949332 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.949414 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.949474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:05.952128 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:05.953950 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:05.961476 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.957080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43765 (local address 127.0.0.1:42706)
0620 00:12:05.957290 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.957300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.957345 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.957705 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.957705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.957707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:05.957791 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.957792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.959163 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.959164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.960203 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.960205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.961126 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.961127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.961214 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.961229 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.961332 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.961391 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":9,"thread_start_us":66,"threads_started":1}
I20260620 00:12:05.961485 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.957289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.98.2:43765)
0620 00:12:05.957476 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.957482 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.957491 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:05.957530 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.957532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.957549 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.957627 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:05.957858 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.959050 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.960296 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.961009 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.961124 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.961192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.961350 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.961391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260620 00:12:05.961727 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.956853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37863 (local address 127.0.0.1:44784)
0620 00:12:05.957246 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.957257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.957352 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.957705 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.957705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.957709 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:05.957829 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.957831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.958745 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.958747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.960142 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.960144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.961447 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.961447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.961531 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.961535 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.961588 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.961646 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":28,"spinlock_wait_cycles":768,"thread_start_us":101,"threads_started":1}
I20260620 00:12:05.961970 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.956979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.1:37863)
0620 00:12:05.957190 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.957196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.957321 (+ 125us) server_negotiation.cc:407] Connection header received
0620 00:12:05.957464 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.957467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.957484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.957553 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:05.957900 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.958614 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.960274 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.961276 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.961355 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.961415 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.961829 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.961874 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260620 00:12:05.962411 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 997a7a21740c4754885584ea9819e611 (DEFAULT_TABLE table=test-table [id=a644cab0e97c4dabbbf714deffaa7593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:05.962807 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997a7a21740c4754885584ea9819e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:05.962674 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 997a7a21740c4754885584ea9819e611 (DEFAULT_TABLE table=test-table [id=a644cab0e97c4dabbbf714deffaa7593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:05.963110 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.956853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39611 (local address 127.0.0.1:58062)
0620 00:12:05.957078 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:05.957129 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:05.957200 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:05.957552 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:05.957555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:05.957557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:05.957642 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.957644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.960294 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.960294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.961292 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:05.961294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.962724 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.962725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:05.962799 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.962806 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:12:05.962994 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 00:12:05.963041 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260620 00:12:05.963122 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997a7a21740c4754885584ea9819e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:05.963171 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:05.956979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.0.98.3:39611)
0620 00:12:05.957190 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:05.957196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:05.957210 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:12:05.957363 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:05.957367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:05.957387 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:05.957462 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:05.957736 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.959035 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.961382 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:05.962624 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:05.962701 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:05.962758 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:05.963034 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:12:05.963088 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260620 00:12:05.964076 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 997a7a21740c4754885584ea9819e611 (DEFAULT_TABLE table=test-table [id=a644cab0e97c4dabbbf714deffaa7593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:05.964445 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997a7a21740c4754885584ea9819e611. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:05.966509 1438 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Bootstrap starting.
I20260620 00:12:05.966711 1439 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Bootstrap starting.
I20260620 00:12:05.967325 1439 tablet_bootstrap.cc:654] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:05.967325 1438 tablet_bootstrap.cc:654] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:05.967929 1439 log.cc:824] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:05.967929 1438 log.cc:824] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:05.968276 1440 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: Bootstrap starting.
I20260620 00:12:05.968820 1439 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: No bootstrap required, opened a new log
I20260620 00:12:05.968900 1439 ts_tablet_manager.cc:1397] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:05.968925 1440 tablet_bootstrap.cc:654] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:05.968945 1438 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: No bootstrap required, opened a new log
I20260620 00:12:05.969013 1438 ts_tablet_manager.cc:1397] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:12:05.969463 1440 log.cc:824] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:05.970180 1440 tablet_bootstrap.cc:492] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: No bootstrap required, opened a new log
I20260620 00:12:05.970252 1440 ts_tablet_manager.cc:1397] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:05.970710 1438 raft_consensus.cc:348] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.970875 1438 raft_consensus.cc:374] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:05.970918 1438 raft_consensus.cc:729] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f376a338604924b440534a8dc062f8, State: Initialized, Role: FOLLOWER
I20260620 00:12:05.971027 1438 consensus_queue.cc:260] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.971071 1439 raft_consensus.cc:348] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.971252 1439 raft_consensus.cc:374] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:05.971297 1439 raft_consensus.cc:729] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d22b0cd49b474646b30f7e47bc0d2c84, State: Initialized, Role: FOLLOWER
I20260620 00:12:05.971342 1438 ts_tablet_manager.cc:1428] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:05.971459 1287 heartbeater.cc:503] Master 127.0.98.62:39035 was elected leader, sending a full tablet report...
I20260620 00:12:05.971436 1439 consensus_queue.cc:260] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.972110 1153 heartbeater.cc:503] Master 127.0.98.62:39035 was elected leader, sending a full tablet report...
I20260620 00:12:05.972469 1439 ts_tablet_manager.cc:1428] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:12:05.972755 1440 raft_consensus.cc:348] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.972869 1440 raft_consensus.cc:374] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:05.972888 1440 raft_consensus.cc:729] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff884c536a74ca6962ce451f2776660, State: Initialized, Role: FOLLOWER
I20260620 00:12:05.973001 1440 consensus_queue.cc:260] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:05.973268 1440 ts_tablet_manager.cc:1428] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:05.973471 1421 heartbeater.cc:503] Master 127.0.98.62:39035 was elected leader, sending a full tablet report...
I20260620 00:12:06.004427 1446 raft_consensus.cc:482] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:06.004637 1446 raft_consensus.cc:504] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:06.005084 1446 leader_election.cc:288] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d22b0cd49b474646b30f7e47bc0d2c84 (127.0.98.1:37863), 83f376a338604924b440534a8dc062f8 (127.0.98.2:43765)
I20260620 00:12:06.008831 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.005339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41427 (local address 127.0.98.1:37863)
0620 00:12:06.005388 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:06.005390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:06.005401 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:06.005456 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:06.005457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:06.005470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:06.005545 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:06.005722 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.006487 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.007986 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.008408 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.008503 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.008513 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:06.008720 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 00:12:06.008762 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:12:06.008827 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.005612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53015 (local address 127.0.98.2:43765)
0620 00:12:06.005671 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:06.005673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:06.005686 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:12:06.005716 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:06.005716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:06.005730 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:06.005782 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:06.005887 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.006559 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.008004 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.008396 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.008456 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.008464 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:06.008705 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 00:12:06.008753 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:12:06.008836 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.005252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37863 (local address 127.0.98.3:41427)
0620 00:12:06.005311 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:06.005326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:06.005389 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:06.005551 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:06.005552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:06.005553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:06.005637 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:06.005639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.006641 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.006642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:06.007882 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:06.007885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.008524 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.008525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:06.008654 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.008658 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:06.008715 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:12:06.008754 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":6}
I20260620 00:12:06.008845 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.005431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43765 (local address 127.0.98.3:53015)
0620 00:12:06.005601 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:06.005612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:06.005666 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:06.005786 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:06.005786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:06.005788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:06.005845 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:06.005846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.006674 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.006675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:06.007882 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:06.007885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.008491 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.008492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:06.008589 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.008594 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:12:06.008696 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:12:06.008741 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":5,"thread_start_us":125,"threads_started":1}
I20260620 00:12:06.009238 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997a7a21740c4754885584ea9819e611" candidate_uuid: "5ff884c536a74ca6962ce451f2776660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f376a338604924b440534a8dc062f8" is_pre_election: true
I20260620 00:12:06.009308 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997a7a21740c4754885584ea9819e611" candidate_uuid: "5ff884c536a74ca6962ce451f2776660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" is_pre_election: true
I20260620 00:12:06.009438 1241 raft_consensus.cc:2454] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff884c536a74ca6962ce451f2776660 in term 0.
I20260620 00:12:06.009445 1107 raft_consensus.cc:2454] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ff884c536a74ca6962ce451f2776660 in term 0.
I20260620 00:12:06.009693 1310 leader_election.cc:302] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff884c536a74ca6962ce451f2776660, d22b0cd49b474646b30f7e47bc0d2c84; no voters:
I20260620 00:12:06.009838 1446 raft_consensus.cc:2781] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:06.009910 1446 raft_consensus.cc:482] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:06.009933 1446 raft_consensus.cc:3037] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:06.010701 1446 raft_consensus.cc:504] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:06.010808 1446 leader_election.cc:288] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [CANDIDATE]: Term 1 election: Requested vote from peers d22b0cd49b474646b30f7e47bc0d2c84 (127.0.98.1:37863), 83f376a338604924b440534a8dc062f8 (127.0.98.2:43765)
I20260620 00:12:06.011032 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997a7a21740c4754885584ea9819e611" candidate_uuid: "5ff884c536a74ca6962ce451f2776660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d22b0cd49b474646b30f7e47bc0d2c84"
I20260620 00:12:06.011077 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997a7a21740c4754885584ea9819e611" candidate_uuid: "5ff884c536a74ca6962ce451f2776660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f376a338604924b440534a8dc062f8"
I20260620 00:12:06.011135 1241 raft_consensus.cc:3037] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:06.011147 1107 raft_consensus.cc:3037] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:06.011932 1241 raft_consensus.cc:2454] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff884c536a74ca6962ce451f2776660 in term 1.
I20260620 00:12:06.011957 1107 raft_consensus.cc:2454] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ff884c536a74ca6962ce451f2776660 in term 1.
I20260620 00:12:06.012145 1310 leader_election.cc:302] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ff884c536a74ca6962ce451f2776660, d22b0cd49b474646b30f7e47bc0d2c84; no voters:
I20260620 00:12:06.012281 1446 raft_consensus.cc:2781] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:06.012461 1446 raft_consensus.cc:686] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff884c536a74ca6962ce451f2776660, State: Running, Role: LEADER
I20260620 00:12:06.012619 1446 consensus_queue.cc:237] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } }
I20260620 00:12:06.013540 961 catalog_manager.cc:5495] T 997a7a21740c4754885584ea9819e611 P 5ff884c536a74ca6962ce451f2776660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ff884c536a74ca6962ce451f2776660 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ff884c536a74ca6962ce451f2776660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d22b0cd49b474646b30f7e47bc0d2c84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ff884c536a74ca6962ce451f2776660" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f376a338604924b440534a8dc062f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43765 } health_report { overall_health: UNKNOWN } } }
W20260620 00:12:06.023799 1288 tablet.cc:2369] T 997a7a21740c4754885584ea9819e611 P 83f376a338604924b440534a8dc062f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:06.029403 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.021009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.98.1:37863)
0620 00:12:06.021081 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:06.021083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:06.021240 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:12:06.021290 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:06.021290 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:06.021299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:06.021350 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:06.021946 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.022422 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.023197 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.023399 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.023456 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.025869 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:06.025881 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:06.025882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:06.025890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:06.028189 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.028571 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.028576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.028576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.028643 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.028844 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.028845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.028846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.029000 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:06.029166 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:06.029211 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 00:12:06.029292 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:06.056670 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.048392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.0.98.1:37863)
0620 00:12:06.048577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:06.048580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:06.048593 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:12:06.048710 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:06.048712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:06.048722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:06.048807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:06.049449 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.049947 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.050719 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.050920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.051022 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.053751 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:06.053754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:06.053754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:06.053763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:06.055418 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.055842 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.055843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.055844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.055904 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.056107 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.056108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.056108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.056209 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:06.056285 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:06.056444 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 00:12:06.056513 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 00:12:06.072659 1154 tablet.cc:2369] T 997a7a21740c4754885584ea9819e611 P d22b0cd49b474646b30f7e47bc0d2c84: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:06.080955 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:06.073140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.0.98.1:37863)
0620 00:12:06.073212 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:06.073214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:06.073282 (+ 68us) server_negotiation.cc:407] Connection header received
0620 00:12:06.073342 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:06.073343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:06.073353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:06.073413 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:06.074031 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.074530 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.075376 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:06.075573 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:06.075642 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:06.078251 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:06.078252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:06.078252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:06.078260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:06.079790 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.080164 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.080165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.080165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.080216 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:06.080395 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:06.080395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:06.080396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:06.080492 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:06.080671 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:06.080792 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:12:06.080860 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 00:12:06.081481 1067 tablet_service.cc:2908] Rejecting scan request for tablet 997a7a21740c4754885584ea9819e611: Uninitialized: clean time has not yet been initialized
I20260620 00:12:06.082515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1024
I20260620 00:12:06.087639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1158
I20260620 00:12:06.092238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1292
I20260620 00:12:06.096479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 930
2026-06-20T00:12:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2089 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[1462](info): set up 2 sockets
Jun 20 00:12:06 dist-test-slave-28bv krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 00:12:07.176800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:07.329227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:07.419894 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:07.420235 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:07.420281 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:07.421819 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:07.421927 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:12:07.421955 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:07.421977 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:07.421999 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:07.422027 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:07.424019 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34373
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:07.424401 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:07.424749 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:07.431746 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:07.431921 1475 server_base.cc:1034] running on GCE node
W20260620 00:12:07.431752 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:07.431773 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:07.432400 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:07.432816 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:07.434060 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781914327433969 us; error 115 us; skew 500 ppm
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:07.442414 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:07.443286 1475 webserver.cc:460] Webserver started at http://127.0.98.62:45391/ using document root <none> and password file <none>
I20260620 00:12:07.443609 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:07.443681 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:07.443850 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:07.445245 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "66d9648203234fb8b0cec6d056fd4f3e"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "a8c6cd78da40ff16cd263408be6f3d72"
server_key_iv: "c0ec3883d1d73fa7783f5a15edee6ce3"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.445690 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "66d9648203234fb8b0cec6d056fd4f3e"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "a8c6cd78da40ff16cd263408be6f3d72"
server_key_iv: "c0ec3883d1d73fa7783f5a15edee6ce3"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.447403 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:07.448589 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.449057 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:07.449165 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "66d9648203234fb8b0cec6d056fd4f3e"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "a8c6cd78da40ff16cd263408be6f3d72"
server_key_iv: "c0ec3883d1d73fa7783f5a15edee6ce3"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.449311 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:07.473937 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:07.476796 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:07.476934 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:07.482539 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34373
I20260620 00:12:07.482581 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34373 every 8 connection(s)
I20260620 00:12:07.483036 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:07.484005 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:07.486112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1475
I20260620 00:12:07.486461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:07.486768 392 external_mini_cluster.cc:1428] Setting key 82ece752f06ad53ce70c1e2294451758
I20260620 00:12:07.488690 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Bootstrap starting.
I20260620 00:12:07.489678 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:07.490043 1553 log.cc:824] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:07.490988 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: No bootstrap required, opened a new log
I20260620 00:12:07.493664 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } }
I20260620 00:12:07.493849 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:07.493877 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d9648203234fb8b0cec6d056fd4f3e, State: Initialized, Role: FOLLOWER
I20260620 00:12:07.494055 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } }
I20260620 00:12:07.494148 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:07.494177 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:07.494256 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:07.495169 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } }
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:07.495324 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d9648203234fb8b0cec6d056fd4f3e; no voters:
I20260620 00:12:07.495560 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:07.495756 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:07.495949 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:07.496145 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [term 1 LEADER]: Becoming Leader. State: Replica: 66d9648203234fb8b0cec6d056fd4f3e, State: Running, Role: LEADER
I20260620 00:12:07.496434 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } }
I20260620 00:12:07.497522 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d9648203234fb8b0cec6d056fd4f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } } }
I20260620 00:12:07.497860 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:07.497676 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d9648203234fb8b0cec6d056fd4f3e. Latest consensus state: current_term: 1 leader_uuid: "66d9648203234fb8b0cec6d056fd4f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d9648203234fb8b0cec6d056fd4f3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34373 } } }
I20260620 00:12:07.498063 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e [sys.catalog]: This master's current role is: LEADER
W20260620 00:12:07.498378 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:07.498442 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:07.498503 1573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:07.498684 1573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:07.500617 1573 catalog_manager.cc:1347] Generated new cluster ID: d517a3cb36554d108276345a1233adae
I20260620 00:12:07.500692 1573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:07.501463 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.487559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.0.98.62:34373)
0620 00:12:07.487887 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:07.487901 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:07.487948 (+ 47us) server_negotiation.cc:407] Connection header received
0620 00:12:07.488461 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:07.488469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:07.488603 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:07.488766 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:07.489676 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.491016 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.491863 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.492721 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.492891 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.496082 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:07.496088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:07.496089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:07.496105 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:07.499287 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.499823 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.499826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.499826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.499958 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.500231 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.500232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.500232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.500598 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:07.500776 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:07.501064 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 00:12:07.501233 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":155,"threads_started":1}
I20260620 00:12:07.528930 1573 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:07.529649 1573 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:07.534024 1573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Generated new TSK 0
I20260620 00:12:07.534272 1573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:07.572341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34373
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:07.665222 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:07.665472 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:07.665501 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:07.666989 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:07.667057 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:07.667080 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:07.667100 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:07.668936 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:07.669227 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:07.669531 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:07.675062 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:07.675072 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:07.675091 1579 server_base.cc:1034] running on GCE node
W20260620 00:12:07.675055 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:07.675529 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:07.675794 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:07.676954 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781914327676926 us; error 38 us; skew 500 ppm
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:07.679986 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:07.680709 1579 webserver.cc:460] Webserver started at http://127.0.98.1:34111/ using document root <none> and password file <none>
I20260620 00:12:07.681041 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:07.681123 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:07.681289 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:07.682613 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "fb36d2464d3305d6f6ba769d8e3241ab"
server_key_iv: "aa64dada97daa6601dd9d559d3b03b5f"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.683048 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "fb36d2464d3305d6f6ba769d8e3241ab"
server_key_iv: "aa64dada97daa6601dd9d559d3b03b5f"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.685298 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 00:12:07.686506 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.686939 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:07.687072 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "fb36d2464d3305d6f6ba769d8e3241ab"
server_key_iv: "aa64dada97daa6601dd9d559d3b03b5f"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.687175 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:07.712143 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:07.714764 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:07.714982 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:07.715598 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:07.715693 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.715758 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:07.715787 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.723769 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36351
I20260620 00:12:07.723875 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36351 every 8 connection(s)
I20260620 00:12:07.724258 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:07.727957 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.716415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43047 (local address 127.0.98.62:34373)
0620 00:12:07.716505 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:07.716507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:07.716771 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:12:07.717448 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:07.717450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:07.717465 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:07.717513 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:07.718331 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.719180 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.721182 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.721533 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.721611 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.725012 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:07.725014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:07.725014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:07.725022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:07.726351 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.726842 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.726844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.726844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.726906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.727204 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.727204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.727205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.727330 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:07.727386 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:07.727793 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 00:12:07.727886 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 00:12:07.728196 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.716407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.1:43047)
0620 00:12:07.716630 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:07.716671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:07.717307 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:07.717662 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:07.717666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:07.717938 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:07.718220 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.718225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.719295 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.719296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.721046 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.721051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.721640 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.721642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.721827 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.722646 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:07.722661 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:07.724891 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:07.726490 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.726497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.726499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.726745 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.726999 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.727000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.727000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.727127 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.727389 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:07.727393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:07.727522 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 00:12:07.727717 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 00:12:07.727927 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260620 00:12:07.728214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1579
I20260620 00:12:07.728322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:07.728525 392 external_mini_cluster.cc:1428] Setting key d11cf86c67192ffcdc905cb7a4186b81
I20260620 00:12:07.729192 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34373
I20260620 00:12:07.730659 1710 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:07.730981 1710 heartbeater.cc:511] Master 127.0.98.62:34373 requested a full tablet report, sending...
I20260620 00:12:07.731564 1506 ts_manager.cc:194] Registered new tserver with Master: 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1:36351)
I20260620 00:12:07.732589 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43047
I20260620 00:12:07.736836 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.730246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53281 (local address 127.0.98.62:34373)
0620 00:12:07.730290 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:07.730292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:07.730369 (+ 77us) server_negotiation.cc:407] Connection header received
0620 00:12:07.730451 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:07.730452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:07.730464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:07.730533 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:07.730939 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.731718 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.732489 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.732744 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.732829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.734055 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:07.734056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:07.734057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:07.734065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:07.735556 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.735955 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.735957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.735957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.736022 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.736242 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.736243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.736244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.736368 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:07.736450 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:07.736642 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 00:12:07.736733 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:12:07.736855 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.730134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.1:53281)
0620 00:12:07.730285 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:07.730298 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:07.730378 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:07.730551 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:07.730552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:07.730777 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:07.730855 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.730858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.731826 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.731827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.732395 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.732396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.733010 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.733011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.733073 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.733621 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:07.733626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:07.733954 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:07.735672 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.735673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.735674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.735864 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.736105 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.736106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.736107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.736175 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.736441 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:07.736442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:07.736495 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:12:07.736615 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:12:07.736736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:07.759693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34373
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:07.841904 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:07.842163 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:07.842211 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:07.843500 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:07.843578 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:07.843613 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:07.843653 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:07.845402 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:07.845703 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:07.845953 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:07.851275 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:07.851307 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:07.851279 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:07.851544 1719 server_base.cc:1034] running on GCE node
I20260620 00:12:07.851753 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:07.852029 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:07.853212 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781914327853198 us; error 30 us; skew 500 ppm
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:07.855751 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:07.856432 1719 webserver.cc:460] Webserver started at http://127.0.98.2:33503/ using document root <none> and password file <none>
I20260620 00:12:07.856722 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:07.856792 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:07.856954 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:07.858018 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "cef4b6c6f6e14b29bef529af373f2edc"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "ea0dc45afabc712dca32cf7f741ed2eb"
server_key_iv: "865fee803bd82caeacf42b81c43d2f63"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.858388 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cef4b6c6f6e14b29bef529af373f2edc"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "ea0dc45afabc712dca32cf7f741ed2eb"
server_key_iv: "865fee803bd82caeacf42b81c43d2f63"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.859789 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:07.860687 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.860900 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:07.861007 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "cef4b6c6f6e14b29bef529af373f2edc"
format_stamp: "Formatted at 2026-06-20 00:12:07 on dist-test-slave-28bv"
server_key: "ea0dc45afabc712dca32cf7f741ed2eb"
server_key_iv: "865fee803bd82caeacf42b81c43d2f63"
server_key_version: "encryptionkey@0"
I20260620 00:12:07.861106 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:07.874154 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:07.876349 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:07.876533 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:07.877179 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:07.877238 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.877280 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:07.877306 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:07.884672 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38275
I20260620 00:12:07.884699 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38275 every 8 connection(s)
Jun 20 00:12:07 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:07.885099 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:07.885288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1719
I20260620 00:12:07.885380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:07.885571 392 external_mini_cluster.cc:1428] Setting key c027ee70d0965b07e018e5555e34f8c1
I20260620 00:12:07.888834 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.877680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34953 (local address 127.0.98.62:34373)
0620 00:12:07.877744 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:07.877746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:07.877954 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:12:07.878630 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:07.878631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:07.878646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:07.878722 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:07.879462 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.880283 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.881726 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.882086 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.882149 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.885773 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:07.885776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:07.885776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:07.885787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:07.887339 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.887805 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.887807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.887807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.887871 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.888136 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.888137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.888137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.888262 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:07.888340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:07.888654 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 00:12:07.888736 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:12:07.889045 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.877645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.2:34953)
0620 00:12:07.877813 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:07.877851 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:07.878471 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:07.878830 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:07.878835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:07.879116 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:07.879327 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.879332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.880404 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.880405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.881580 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.881584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.882170 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.882171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.882352 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.883596 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:07.883608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:07.885637 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:07.887449 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.887457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.887459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.887707 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.887941 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.887942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.887942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.888064 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.888330 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:07.888336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:07.888484 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 00:12:07.888635 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 00:12:07.888804 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260620 00:12:07.889518 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34373
I20260620 00:12:07.889598 1850 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:07.889793 1850 heartbeater.cc:511] Master 127.0.98.62:34373 requested a full tablet report, sending...
I20260620 00:12:07.890162 1506 ts_manager.cc:194] Registered new tserver with Master: cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275)
I20260620 00:12:07.890658 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34953
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:12:07.899123 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.890701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45179 (local address 127.0.98.62:34373)
0620 00:12:07.890735 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:07.890737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:07.892742 (+ 2005us) server_negotiation.cc:407] Connection header received
0620 00:12:07.892819 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:07.892820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:07.892831 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:07.892942 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:07.893286 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.893827 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.894598 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.894779 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.894831 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.896006 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:07.896008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:07.896008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:07.896016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:07.897839 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.898261 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.898263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.898263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.898320 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:07.898545 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:07.898546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:07.898547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:07.898704 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:07.898782 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:07.898936 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:12:07.899018 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:12:07.899139 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:07.890645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.2:45179)
0620 00:12:07.892663 (+ 2018us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:07.892675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:07.892758 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:07.892952 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:07.892952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:07.893140 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:07.893191 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.893193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.893919 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.893921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.894481 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:07.894483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:07.894851 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:07.894852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:07.894912 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:07.895471 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:07.895477 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:07.895842 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:07.897924 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.897925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.897926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.898189 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.898406 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:07.898406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:07.898407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:07.898467 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:07.898773 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:07.898773 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:07.898820 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 00:12:07.898909 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:12:07.899019 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:07.916496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34373
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:08.001008 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:08.001228 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:08.001255 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:08.002735 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:08.002791 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:08.002815 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:08.002832 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:08.004384 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:08.004673 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:08.004940 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:08.010273 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:08.010414 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:08.010294 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:08.010507 1859 server_base.cc:1034] running on GCE node
I20260620 00:12:08.010732 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:08.011018 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:08.012178 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781914328012166 us; error 26 us; skew 500 ppm
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:08.015077 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:08.015635 1859 webserver.cc:460] Webserver started at http://127.0.98.3:39701/ using document root <none> and password file <none>
I20260620 00:12:08.015887 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:08.015960 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:08.016121 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:08.017482 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "382cc7569c504d4f9225c11b79130355"
format_stamp: "Formatted at 2026-06-20 00:12:08 on dist-test-slave-28bv"
server_key: "5ee71dfefcb0a7604b2ba5c4606c11cc"
server_key_iv: "b9e5530ca27ed8744a91f444d85a0d1a"
server_key_version: "encryptionkey@0"
I20260620 00:12:08.017876 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "382cc7569c504d4f9225c11b79130355"
format_stamp: "Formatted at 2026-06-20 00:12:08 on dist-test-slave-28bv"
server_key: "5ee71dfefcb0a7604b2ba5c4606c11cc"
server_key_iv: "b9e5530ca27ed8744a91f444d85a0d1a"
server_key_version: "encryptionkey@0"
I20260620 00:12:08.019495 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:12:08.020412 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.020640 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:08.020752 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "382cc7569c504d4f9225c11b79130355"
format_stamp: "Formatted at 2026-06-20 00:12:08 on dist-test-slave-28bv"
server_key: "5ee71dfefcb0a7604b2ba5c4606c11cc"
server_key_iv: "b9e5530ca27ed8744a91f444d85a0d1a"
server_key_version: "encryptionkey@0"
I20260620 00:12:08.020834 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:08.030952 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:08.033617 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:08.033794 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:08.034475 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:08.034571 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.034610 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:08.034641 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.041940 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35103
I20260620 00:12:08.042117 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35103 every 8 connection(s)
I20260620 00:12:08.042361 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:08.046389 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.035063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57607 (local address 127.0.98.62:34373)
0620 00:12:08.035106 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.035108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.035347 (+ 239us) server_negotiation.cc:407] Connection header received
0620 00:12:08.035952 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.035953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.035966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.036022 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.037014 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.037893 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.039656 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.039907 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.039968 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.043129 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.043131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.043131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.043139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.044883 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.045382 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.045383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.045384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.045449 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.045710 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.045711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.045712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.045840 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.045916 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.046204 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.046289 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:08.046604 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.035049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.3:57607)
0620 00:12:08.035221 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.035257 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.035813 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.036105 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.036110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.036501 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:08.036865 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.036915 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.038012 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.038013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.039544 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.039548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.040001 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.040002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.040173 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.041013 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:08.041026 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:08.043013 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:08.045017 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.045027 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.045029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.045286 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.045518 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.045519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.045519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.045641 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.045908 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:08.045914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:08.046073 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.046185 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.046357 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260620 00:12:08.047077 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34373
I20260620 00:12:08.047170 1990 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:08.047408 1990 heartbeater.cc:511] Master 127.0.98.62:34373 requested a full tablet report, sending...
I20260620 00:12:08.047786 1507 ts_manager.cc:194] Registered new tserver with Master: 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103)
I20260620 00:12:08.048296 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57607
I20260620 00:12:08.051894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 1859
I20260620 00:12:08.052011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:08.052212 392 external_mini_cluster.cc:1428] Setting key 74cd37d4d69a8d4a61018fee4a463be6
I20260620 00:12:08.053205 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:12:08.054145 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.048206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53687 (local address 127.0.98.62:34373)
0620 00:12:08.048246 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.048247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.048342 (+ 95us) server_negotiation.cc:407] Connection header received
0620 00:12:08.048401 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.048402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.048417 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.048509 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.048896 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.049349 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.050094 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.050278 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.050344 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.051298 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.051300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.051300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.051306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.053033 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.053459 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.053460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.053461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.053518 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.053706 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.053707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.053707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.053807 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.053858 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.054028 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.054084 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:12:08.054292 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.048143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.98.3:53687)
0620 00:12:08.048265 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.048275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.048334 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.048515 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.048516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.048748 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:08.048820 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.048822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.049452 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.049453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.050000 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.050002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.050382 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.050383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.050439 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.050918 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:08.050922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:08.051210 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:08.053117 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.053118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.053118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.053378 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.053580 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.053581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.053581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.053645 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.053872 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:08.053873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:08.053926 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.054010 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.054127 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:08.089284 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.080572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.98.62:34373)
0620 00:12:08.080630 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.080632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.080644 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:08.080683 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.080684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.080694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.080767 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.081799 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.082510 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.083875 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.084073 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.084136 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.086763 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.086765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.086765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.086774 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.088134 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.088496 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.088497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.088498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.088619 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.088817 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.088818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.088818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.088943 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.089028 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.089135 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.089201 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:12:08.091517 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:08.093312 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:08.100178 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.096053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.0.98.3:35103)
0620 00:12:08.096208 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.096215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.096226 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:08.096268 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.096270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.096324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.096652 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.096711 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.097544 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099056 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.099697 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.099865 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.100017 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.100056 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260620 00:12:08.100183 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.095790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35103 (local address 127.0.0.1:40634)
0620 00:12:08.095994 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.096003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.096062 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.096439 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.096443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.096446 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.096528 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.096531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.097630 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.097631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.098963 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.098965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.099796 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.099887 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.099901 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.100003 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.100060 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":38,"thread_start_us":103,"threads_started":1}
I20260620 00:12:08.100370 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.095852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.0.98.2:38275)
0620 00:12:08.096252 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.096258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.096271 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:12:08.096316 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.096318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.096334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.096412 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.096679 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.097841 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099096 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.099765 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099835 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.099916 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.100224 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.100271 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":67,"threads_started":1}
I20260620 00:12:08.100404 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.096045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36351 (local address 127.0.0.1:57408)
0620 00:12:08.096287 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.096297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.096349 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.096913 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.096913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.096916 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.096950 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.096951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.098225 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.098226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.099213 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.099215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.100172 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.100173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.100235 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.100238 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.100285 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.100318 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":71,"threads_started":1}
I20260620 00:12:08.100503 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.095751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38275 (local address 127.0.0.1:46126)
0620 00:12:08.095950 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.095991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.096062 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.096507 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.096508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.096510 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.096597 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.096599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.097929 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.097930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.099013 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.099014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.099952 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.100026 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.100030 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.100078 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.100120 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260620 00:12:08.100802 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.096306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.0.98.1:36351)
0620 00:12:08.096519 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.096523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.096637 (+ 114us) server_negotiation.cc:407] Connection header received
0620 00:12:08.096753 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.096756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.096774 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.096849 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.097032 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.098148 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.099304 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.100003 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.100071 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.100140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.100645 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.100694 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
I20260620 00:12:08.101327 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e4067a6db89148c1aa33db4a75aee381 (DEFAULT_TABLE table=test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.101356 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e4067a6db89148c1aa33db4a75aee381 (DEFAULT_TABLE table=test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.101358 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e4067a6db89148c1aa33db4a75aee381 (DEFAULT_TABLE table=test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.101637 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4067a6db89148c1aa33db4a75aee381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.101686 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4067a6db89148c1aa33db4a75aee381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.101687 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4067a6db89148c1aa33db4a75aee381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.105024 2013 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Bootstrap starting.
I20260620 00:12:08.105621 2014 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Bootstrap starting.
I20260620 00:12:08.105801 2013 tablet_bootstrap.cc:654] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.106146 2013 log.cc:824] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:08.106278 2014 tablet_bootstrap.cc:654] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.106575 2014 log.cc:824] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:08.106941 2015 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Bootstrap starting.
I20260620 00:12:08.106995 2013 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: No bootstrap required, opened a new log
I20260620 00:12:08.107052 2013 ts_tablet_manager.cc:1397] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:08.107395 2014 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: No bootstrap required, opened a new log
I20260620 00:12:08.107475 2014 ts_tablet_manager.cc:1397] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:08.107576 2015 tablet_bootstrap.cc:654] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.107885 2015 log.cc:824] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:08.108898 2015 tablet_bootstrap.cc:492] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: No bootstrap required, opened a new log
I20260620 00:12:08.109004 2015 ts_tablet_manager.cc:1397] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:08.109481 2014 raft_consensus.cc:348] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.109648 2014 raft_consensus.cc:374] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.109699 2014 raft_consensus.cc:729] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c4589c8ebd4f92adb8d04df3a0a3ba, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.109851 2014 consensus_queue.cc:260] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.109778 2013 raft_consensus.cc:348] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.109941 2013 raft_consensus.cc:374] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.109988 2013 raft_consensus.cc:729] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef4b6c6f6e14b29bef529af373f2edc, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.110139 2014 ts_tablet_manager.cc:1428] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:08.110137 2013 consensus_queue.cc:260] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.110287 1710 heartbeater.cc:503] Master 127.0.98.62:34373 was elected leader, sending a full tablet report...
I20260620 00:12:08.110427 2013 ts_tablet_manager.cc:1428] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:08.110466 1850 heartbeater.cc:503] Master 127.0.98.62:34373 was elected leader, sending a full tablet report...
I20260620 00:12:08.111375 2015 raft_consensus.cc:348] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.111567 2015 raft_consensus.cc:374] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.111603 2015 raft_consensus.cc:729] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382cc7569c504d4f9225c11b79130355, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.111721 2015 consensus_queue.cc:260] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.112017 2015 ts_tablet_manager.cc:1428] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:08.112043 1990 heartbeater.cc:503] Master 127.0.98.62:34373 was elected leader, sending a full tablet report...
W20260620 00:12:08.135830 1851 tablet.cc:2369] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:12:08.225196 1711 tablet.cc:2369] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:08.282533 2019 raft_consensus.cc:482] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:08.282658 2019 raft_consensus.cc:504] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.283061 2019 leader_election.cc:288] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103), cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275)
I20260620 00:12:08.286234 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.283547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41297 (local address 127.0.98.2:38275)
0620 00:12:08.283606 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.283608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.283619 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:08.283715 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.283716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.283727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.283812 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.284011 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.284479 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.285501 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.285866 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.285961 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.285972 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.286120 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.286157 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:12:08.286257 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.283329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38275 (local address 127.0.98.1:41297)
0620 00:12:08.283546 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.283558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.283632 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.283819 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.283820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.283822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.283928 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.283930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.284611 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.284612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.285398 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.285401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.285963 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.285964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.286042 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.286045 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.286114 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.286157 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260620 00:12:08.286434 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.283553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51547 (local address 127.0.98.3:35103)
0620 00:12:08.283623 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.283625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.283634 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:08.283663 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.283663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.283677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.283757 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.283955 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.284736 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.285672 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.286054 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.286147 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.286157 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.286317 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.286363 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:12:08.286525 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.283327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35103 (local address 127.0.98.1:51547)
0620 00:12:08.283503 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.283517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.283576 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.283760 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.283762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.283764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.283875 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.283876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.284844 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.284845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.285577 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.285579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.286153 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.286154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.286220 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.286223 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.286301 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.286424 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
I20260620 00:12:08.286732 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4067a6db89148c1aa33db4a75aee381" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc" is_pre_election: true
I20260620 00:12:08.286870 1803 raft_consensus.cc:2454] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 0.
I20260620 00:12:08.287139 1595 leader_election.cc:302] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c4589c8ebd4f92adb8d04df3a0a3ba, cef4b6c6f6e14b29bef529af373f2edc; no voters:
I20260620 00:12:08.287298 2019 raft_consensus.cc:2781] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:08.287357 2019 raft_consensus.cc:482] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:08.287380 2019 raft_consensus.cc:3037] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.287309 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4067a6db89148c1aa33db4a75aee381" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382cc7569c504d4f9225c11b79130355" is_pre_election: true
I20260620 00:12:08.287534 1944 raft_consensus.cc:2454] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 0.
I20260620 00:12:08.288194 2019 raft_consensus.cc:504] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.288354 2019 leader_election.cc:288] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 election: Requested vote from peers 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103), cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275)
I20260620 00:12:08.288604 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4067a6db89148c1aa33db4a75aee381" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc"
I20260620 00:12:08.288633 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4067a6db89148c1aa33db4a75aee381" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382cc7569c504d4f9225c11b79130355"
I20260620 00:12:08.288707 1803 raft_consensus.cc:3037] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.288733 1944 raft_consensus.cc:3037] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.289543 1944 raft_consensus.cc:2454] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 1.
I20260620 00:12:08.289551 1803 raft_consensus.cc:2454] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 1.
I20260620 00:12:08.289729 1595 leader_election.cc:302] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c4589c8ebd4f92adb8d04df3a0a3ba, cef4b6c6f6e14b29bef529af373f2edc; no voters:
I20260620 00:12:08.289840 2019 raft_consensus.cc:2781] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:08.289954 2019 raft_consensus.cc:686] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 LEADER]: Becoming Leader. State: Replica: 29c4589c8ebd4f92adb8d04df3a0a3ba, State: Running, Role: LEADER
I20260620 00:12:08.290087 2019 consensus_queue.cc:237] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } }
I20260620 00:12:08.290851 1506 catalog_manager.cc:5495] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } health_report { overall_health: HEALTHY } } }
W20260620 00:12:08.293044 1991 tablet.cc:2369] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:08.328299 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.325509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.0.98.1:36351)
0620 00:12:08.325590 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.325592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.325603 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:08.325684 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.325685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.325703 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.325769 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:08.325955 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.326475 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.327371 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.327566 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.327629 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.327894 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.328002 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.328072 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.328152 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.328172 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:12:08.330293 1944 raft_consensus.cc:1261] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Refusing update from remote peer 29c4589c8ebd4f92adb8d04df3a0a3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.330439 1803 raft_consensus.cc:1261] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Refusing update from remote peer 29c4589c8ebd4f92adb8d04df3a0a3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.330742 2024 consensus_queue.cc:1035] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.330909 2019 consensus_queue.cc:1035] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.333442 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298721088838049792 to 7298721088676925440. Current Snapshot: MvccSnapshot[applied={T|T < 7298721088838049792}]
I20260620 00:12:08.334307 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298721088838049792 to 7298721088676925440. Current Snapshot: MvccSnapshot[applied={T|T < 7298721088838049792}]
I20260620 00:12:08.334429 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298721088838049792 to 7298721088676925440. Current Snapshot: MvccSnapshot[applied={T|T < 7298721088838049792 or (T in {7298721088838049792})}]
I20260620 00:12:08.337185 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:08.337281 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:08.338045 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Sending DeleteTablet for 3 replicas of tablet e4067a6db89148c1aa33db4a75aee381
I20260620 00:12:08.338378 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet e4067a6db89148c1aa33db4a75aee381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40634
I20260620 00:12:08.338490 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet e4067a6db89148c1aa33db4a75aee381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260620 00:12:08.338671 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet e4067a6db89148c1aa33db4a75aee381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57408
I20260620 00:12:08.338711 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:08.338852 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:08.338933 2037 tablet_replica.cc:330] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: stopping tablet replica
I20260620 00:12:08.339038 2037 raft_consensus.cc:2229] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:08.339161 2037 raft_consensus.cc:2258] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.339295 2038 tablet_replica.cc:330] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: stopping tablet replica
I20260620 00:12:08.339602 2036 tablet_replica.cc:330] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: stopping tablet replica
I20260620 00:12:08.339751 2038 raft_consensus.cc:2229] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:08.339928 2038 raft_consensus.cc:2258] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.340121 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb (DEFAULT_TABLE table=test-table [id=8f2ba5faa06845009b733696b8387f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.340173 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb (DEFAULT_TABLE table=test-table [id=8f2ba5faa06845009b733696b8387f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.340255 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816003de0ba34f03a6152b0fbc57b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.340298 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816003de0ba34f03a6152b0fbc57b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.340277 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb (DEFAULT_TABLE table=test-table [id=8f2ba5faa06845009b733696b8387f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:08.340372 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816003de0ba34f03a6152b0fbc57b4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.341686 2015 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Bootstrap starting.
I20260620 00:12:08.341684 2013 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Bootstrap starting.
I20260620 00:12:08.341692 2014 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Bootstrap starting.
I20260620 00:12:08.342036 2036 raft_consensus.cc:2229] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:08.342141 2013 tablet_bootstrap.cc:654] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.342144 2014 tablet_bootstrap.cc:654] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.342141 2015 tablet_bootstrap.cc:654] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.343222 2038 ts_tablet_manager.cc:1905] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.343432 2015 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: No bootstrap required, opened a new log
I20260620 00:12:08.343508 2015 ts_tablet_manager.cc:1397] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:08.343665 2013 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: No bootstrap required, opened a new log
I20260620 00:12:08.343701 2013 ts_tablet_manager.cc:1397] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:08.343654 2015 raft_consensus.cc:348] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.343735 2015 raft_consensus.cc:374] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.343757 2015 raft_consensus.cc:729] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382cc7569c504d4f9225c11b79130355, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.343828 2015 consensus_queue.cc:260] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.343919 2015 ts_tablet_manager.cc:1428] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.343808 2013 raft_consensus.cc:348] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.344005 2013 raft_consensus.cc:374] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.344045 2013 raft_consensus.cc:729] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef4b6c6f6e14b29bef529af373f2edc, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.344099 2013 consensus_queue.cc:260] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.344200 2013 ts_tablet_manager.cc:1428] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:08.343416 2036 raft_consensus.cc:2258] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.345178 2037 ts_tablet_manager.cc:1905] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.345510 2036 ts_tablet_manager.cc:1905] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.347090 2037 ts_tablet_manager.cc:1918] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:08.347180 2037 log.cc:1196] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/e4067a6db89148c1aa33db4a75aee381
I20260620 00:12:08.347545 2037 ts_tablet_manager.cc:1939] T e4067a6db89148c1aa33db4a75aee381 P cef4b6c6f6e14b29bef529af373f2edc: Deleting consensus metadata
I20260620 00:12:08.347133 2036 ts_tablet_manager.cc:1918] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:08.347904 2036 log.cc:1196] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/e4067a6db89148c1aa33db4a75aee381
I20260620 00:12:08.347966 2038 ts_tablet_manager.cc:1918] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:08.347980 1491 catalog_manager.cc:4841] TS cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275): tablet e4067a6db89148c1aa33db4a75aee381 (table test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]) successfully deleted
I20260620 00:12:08.348052 2038 log.cc:1196] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/e4067a6db89148c1aa33db4a75aee381
I20260620 00:12:08.348340 2038 ts_tablet_manager.cc:1939] T e4067a6db89148c1aa33db4a75aee381 P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting consensus metadata
I20260620 00:12:08.348407 2036 ts_tablet_manager.cc:1939] T e4067a6db89148c1aa33db4a75aee381 P 382cc7569c504d4f9225c11b79130355: Deleting consensus metadata
I20260620 00:12:08.348871 1492 catalog_manager.cc:4841] TS 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103): tablet e4067a6db89148c1aa33db4a75aee381 (table test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]) successfully deleted
I20260620 00:12:08.348940 1492 catalog_manager.cc:4841] TS 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1:36351): tablet e4067a6db89148c1aa33db4a75aee381 (table test-table [id=13426ff7c63d40e687ada4fe2e0c1b68]) successfully deleted
I20260620 00:12:08.353029 2014 tablet_bootstrap.cc:492] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: No bootstrap required, opened a new log
I20260620 00:12:08.353088 2014 ts_tablet_manager.cc:1397] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260620 00:12:08.353303 2014 raft_consensus.cc:348] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.353402 2014 raft_consensus.cc:374] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.353420 2014 raft_consensus.cc:729] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c4589c8ebd4f92adb8d04df3a0a3ba, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.353487 2014 consensus_queue.cc:260] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.353605 2014 ts_tablet_manager.cc:1428] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.409894 2024 raft_consensus.cc:482] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:08.409998 2024 raft_consensus.cc:504] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.410198 2024 leader_election.cc:288] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275), 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103)
I20260620 00:12:08.410476 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816003de0ba34f03a6152b0fbc57b4cb" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc" is_pre_election: true
I20260620 00:12:08.410480 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816003de0ba34f03a6152b0fbc57b4cb" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382cc7569c504d4f9225c11b79130355" is_pre_election: true
I20260620 00:12:08.410589 1803 raft_consensus.cc:2454] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 0.
I20260620 00:12:08.410594 1944 raft_consensus.cc:2454] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 0.
I20260620 00:12:08.410745 1595 leader_election.cc:302] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c4589c8ebd4f92adb8d04df3a0a3ba, cef4b6c6f6e14b29bef529af373f2edc; no voters:
I20260620 00:12:08.410859 2024 raft_consensus.cc:2781] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:08.410883 2024 raft_consensus.cc:482] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:08.410897 2024 raft_consensus.cc:3037] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.411578 2024 raft_consensus.cc:504] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.411723 2024 leader_election.cc:288] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 election: Requested vote from peers cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275), 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103)
I20260620 00:12:08.411895 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816003de0ba34f03a6152b0fbc57b4cb" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc"
I20260620 00:12:08.411926 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816003de0ba34f03a6152b0fbc57b4cb" candidate_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382cc7569c504d4f9225c11b79130355"
I20260620 00:12:08.411989 1803 raft_consensus.cc:3037] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.412009 1944 raft_consensus.cc:3037] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.412746 1803 raft_consensus.cc:2454] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 1.
I20260620 00:12:08.412894 1944 raft_consensus.cc:2454] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c4589c8ebd4f92adb8d04df3a0a3ba in term 1.
I20260620 00:12:08.412927 1595 leader_election.cc:302] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c4589c8ebd4f92adb8d04df3a0a3ba, cef4b6c6f6e14b29bef529af373f2edc; no voters:
I20260620 00:12:08.413040 2024 raft_consensus.cc:2781] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:08.413084 2024 raft_consensus.cc:686] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 LEADER]: Becoming Leader. State: Replica: 29c4589c8ebd4f92adb8d04df3a0a3ba, State: Running, Role: LEADER
I20260620 00:12:08.413146 2024 consensus_queue.cc:237] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.413901 1506 catalog_manager.cc:5495] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:08.455662 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.445735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.0.98.62:34373)
0620 00:12:08.445796 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.445799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.445913 (+ 114us) server_negotiation.cc:407] Connection header received
0620 00:12:08.445958 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.445959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.445972 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.446066 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.446669 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.447606 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.448769 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.449151 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.449248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.452204 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.452206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.452206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.452215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.454144 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.454752 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.454754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.454754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.454821 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.455016 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.455017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.455017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.455161 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.455228 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.455458 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.455556 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:12:08.455833 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.445646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34373 (local address 127.0.0.1:43568)
0620 00:12:08.445814 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.445828 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.445909 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.446089 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.446090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.446494 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:08.446566 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.446568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.447932 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.447934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.448657 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.448660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.449272 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.449274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.449329 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.450104 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:08.450109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:08.452082 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:08.454251 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.454260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.454262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.454676 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.454894 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:08.454894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:08.454895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:08.454955 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:08.455245 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:08.455252 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:08.455303 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.455426 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.455660 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":95,"thread_start_us":105,"threads_started":1}
I20260620 00:12:08.458410 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43568:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:12:08.458657 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:08.460124 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 856a36c165904a61ad8f29e16299cbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f3e355d65d34df9a8879e8a65bb5724]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:08.460114 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 856a36c165904a61ad8f29e16299cbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f3e355d65d34df9a8879e8a65bb5724]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:08.460266 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856a36c165904a61ad8f29e16299cbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.460268 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856a36c165904a61ad8f29e16299cbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.461019 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 856a36c165904a61ad8f29e16299cbcf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f3e355d65d34df9a8879e8a65bb5724]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:08.461181 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 856a36c165904a61ad8f29e16299cbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:08.462875 2013 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: Bootstrap starting.
I20260620 00:12:08.462880 2015 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: Bootstrap starting.
I20260620 00:12:08.463222 2013 tablet_bootstrap.cc:654] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.463765 2015 tablet_bootstrap.cc:654] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.463837 2013 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: No bootstrap required, opened a new log
I20260620 00:12:08.463874 2013 ts_tablet_manager.cc:1397] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:08.464015 2013 raft_consensus.cc:348] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.464113 2013 raft_consensus.cc:374] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.464138 2013 raft_consensus.cc:729] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef4b6c6f6e14b29bef529af373f2edc, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.464187 2013 consensus_queue.cc:260] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.464298 2013 ts_tablet_manager.cc:1428] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:08.464501 2020 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.464636 2020 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.465592 2014 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: Bootstrap starting.
I20260620 00:12:08.465974 2014 tablet_bootstrap.cc:654] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:08.466099 2015 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: No bootstrap required, opened a new log
I20260620 00:12:08.466148 2015 ts_tablet_manager.cc:1397] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:12:08.466318 2015 raft_consensus.cc:348] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.466380 2015 raft_consensus.cc:374] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.466398 2015 raft_consensus.cc:729] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382cc7569c504d4f9225c11b79130355, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.466449 2015 consensus_queue.cc:260] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.466531 2015 ts_tablet_manager.cc:1428] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:08.466708 2014 tablet_bootstrap.cc:492] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: No bootstrap required, opened a new log
I20260620 00:12:08.466678 2021 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.466750 2014 ts_tablet_manager.cc:1397] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:08.466746 2021 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.466833 2014 raft_consensus.cc:348] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.466940 2014 raft_consensus.cc:374] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:08.466967 2014 raft_consensus.cc:729] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c4589c8ebd4f92adb8d04df3a0a3ba, State: Initialized, Role: FOLLOWER
I20260620 00:12:08.467015 2014 consensus_queue.cc:260] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.467109 2014 ts_tablet_manager.cc:1428] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:08.467216 2024 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.467284 2024 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.532117 2021 raft_consensus.cc:482] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:08.532238 2021 raft_consensus.cc:504] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.532691 2021 leader_election.cc:288] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1:36351), cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275)
I20260620 00:12:08.535596 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.532927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38275 (local address 127.0.98.3:54891)
0620 00:12:08.533002 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.533015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.533066 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.533230 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.533232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.533234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.533305 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.533307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.533957 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.533958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.534741 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.534743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.535308 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.535310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.535387 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.535397 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.535472 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.535520 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 00:12:08.535596 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.533013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54891 (local address 127.0.98.2:38275)
0620 00:12:08.533061 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.533064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.533075 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:08.533134 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.533135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.533143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.533226 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.533372 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.533874 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.534849 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.535219 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.535315 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.535326 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.535480 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.535523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:12:08.535929 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856a36c165904a61ad8f29e16299cbcf" candidate_uuid: "382cc7569c504d4f9225c11b79130355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc" is_pre_election: true
I20260620 00:12:08.536028 1803 raft_consensus.cc:2454] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382cc7569c504d4f9225c11b79130355 in term 0.
I20260620 00:12:08.536204 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.532994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55619 (local address 127.0.98.1:36351)
0620 00:12:08.533041 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.533044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.533281 (+ 237us) server_negotiation.cc:407] Connection header received
0620 00:12:08.533345 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.533346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.533355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.533400 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:08.533604 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.534095 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.535340 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.535806 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.535880 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.535890 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.536085 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.536137 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:08.536217 1875 leader_election.cc:302] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382cc7569c504d4f9225c11b79130355, cef4b6c6f6e14b29bef529af373f2edc; no voters:
I20260620 00:12:08.536336 2021 raft_consensus.cc:2781] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:08.536374 2021 raft_consensus.cc:482] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:08.536412 2021 raft_consensus.cc:3037] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.536515 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.532927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36351 (local address 127.0.98.3:55619)
0620 00:12:08.533205 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.533213 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.533289 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.533450 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.533450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.533453 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:08.533534 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.533535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.534187 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.534188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.535223 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.535232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.535890 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.535891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.535970 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.535973 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.536301 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.536350 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":158,"thread_start_us":80,"threads_started":1}
I20260620 00:12:08.537165 2021 raft_consensus.cc:504] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.537302 2021 leader_election.cc:288] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [CANDIDATE]: Term 1 election: Requested vote from peers 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1:36351), cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275)
I20260620 00:12:08.537465 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856a36c165904a61ad8f29e16299cbcf" candidate_uuid: "382cc7569c504d4f9225c11b79130355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef4b6c6f6e14b29bef529af373f2edc"
I20260620 00:12:08.537552 1803 raft_consensus.cc:3037] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.537516 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856a36c165904a61ad8f29e16299cbcf" candidate_uuid: "382cc7569c504d4f9225c11b79130355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" is_pre_election: true
I20260620 00:12:08.537536 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "856a36c165904a61ad8f29e16299cbcf" candidate_uuid: "382cc7569c504d4f9225c11b79130355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba"
I20260620 00:12:08.537660 1664 raft_consensus.cc:2454] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382cc7569c504d4f9225c11b79130355 in term 0.
I20260620 00:12:08.538242 1803 raft_consensus.cc:2454] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382cc7569c504d4f9225c11b79130355 in term 1.
I20260620 00:12:08.538446 1875 leader_election.cc:302] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 382cc7569c504d4f9225c11b79130355, cef4b6c6f6e14b29bef529af373f2edc; no voters: 29c4589c8ebd4f92adb8d04df3a0a3ba
I20260620 00:12:08.538543 2021 raft_consensus.cc:2781] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:08.538666 2021 raft_consensus.cc:686] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [term 1 LEADER]: Becoming Leader. State: Replica: 382cc7569c504d4f9225c11b79130355, State: Running, Role: LEADER
I20260620 00:12:08.538762 2021 consensus_queue.cc:237] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } }
I20260620 00:12:08.539136 2053 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: TxnStatusTablet state changed. Reason: New leader 382cc7569c504d4f9225c11b79130355. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.539244 2053 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:08.539486 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:12:08.539573 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:12:08.539529 1507 catalog_manager.cc:5495] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 382cc7569c504d4f9225c11b79130355 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } health_report { overall_health: HEALTHY } } }
I20260620 00:12:08.545567 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.542889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.0.98.3:35103)
0620 00:12:08.542950 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.542952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.542982 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:12:08.543078 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.543078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.543094 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.543152 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:08.543305 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.543797 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.544659 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.544877 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.544964 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.545179 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.545282 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.545405 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.545465 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.545488 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:12:08.545585 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.542833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35103 (local address 127.0.0.1:52278)
0620 00:12:08.542936 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.542949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.543001 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.543167 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.543168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.543169 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:08.543218 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.543220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.543885 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.543886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.544468 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.544470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.544976 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.544977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.545057 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.545072 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.545392 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.545402 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.545459 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.545515 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:12:08.546765 1803 raft_consensus.cc:1261] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Refusing update from remote peer 382cc7569c504d4f9225c11b79130355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.546808 1663 raft_consensus.cc:3037] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:08.547024 2053 consensus_queue.cc:1035] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.547632 2020 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: TxnStatusTablet state changed. Reason: New leader 382cc7569c504d4f9225c11b79130355. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.547732 2020 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.547912 1663 raft_consensus.cc:1261] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Refusing update from remote peer 382cc7569c504d4f9225c11b79130355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.548169 2053 consensus_queue.cc:1035] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.549002 2020 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.549103 2020 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P cef4b6c6f6e14b29bef529af373f2edc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.549074 2053 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.549146 2053 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:08.549983 2024 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: TxnStatusTablet state changed. Reason: New leader 382cc7569c504d4f9225c11b79130355. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.550064 2024 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.555704 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.553043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56343 (local address 127.0.98.3:35103)
0620 00:12:08.553088 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.553090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.553193 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:12:08.553296 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.553296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.553307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.553379 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:08.553509 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.554105 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.555018 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.555226 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.555309 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.555474 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.555509 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.555556 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.555612 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.555633 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:12:08.555734 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.552978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35103 (local address 127.0.98.1:56343)
0620 00:12:08.553116 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.553127 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.553210 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.553383 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.553384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.553386 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:08.553434 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.553435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.554183 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.554184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.554937 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.554940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.555311 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.555312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.555380 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.555394 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.555555 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.555557 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.555608 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.555657 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260620 00:12:08.558065 1944 raft_consensus.cc:1261] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Refusing update from remote peer 29c4589c8ebd4f92adb8d04df3a0a3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.558087 1803 raft_consensus.cc:1261] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Refusing update from remote peer 29c4589c8ebd4f92adb8d04df3a0a3ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:08.558347 2024 consensus_queue.cc:1035] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.558488 2024 consensus_queue.cc:1035] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:08.561097 2053 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.561208 2053 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 382cc7569c504d4f9225c11b79130355: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:08.561844 2019 tablet_replica.cc:439] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "382cc7569c504d4f9225c11b79130355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c4589c8ebd4f92adb8d04df3a0a3ba" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36351 } } peers { permanent_uuid: "cef4b6c6f6e14b29bef529af373f2edc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38275 } } peers { permanent_uuid: "382cc7569c504d4f9225c11b79130355" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35103 } } }
I20260620 00:12:08.561951 2019 tablet_replica.cc:442] T 856a36c165904a61ad8f29e16299cbcf P 29c4589c8ebd4f92adb8d04df3a0a3ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:08.567778 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.564504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56727 (local address 127.0.98.1:36351)
0620 00:12:08.564574 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.564577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.564694 (+ 117us) server_negotiation.cc:407] Connection header received
0620 00:12:08.564785 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.564787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.564801 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.564865 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:08.565053 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.565803 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.566730 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.567047 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.567116 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.567469 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.567505 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.567559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.567669 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.567696 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:08.567819 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.564429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36351 (local address 127.0.98.3:56727)
0620 00:12:08.564623 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:08.564635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:08.564724 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:08.564867 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:08.564868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:08.564869 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:08.564972 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.564974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.565898 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.565899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.566641 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:08.566644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.567156 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.567158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:08.567249 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.567266 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:08.567585 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:08.567588 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:08.567654 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:12:08.567703 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260620 00:12:08.577441 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:08.577559 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:08.578212 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66d9648203234fb8b0cec6d056fd4f3e: Sending DeleteTablet for 3 replicas of tablet 816003de0ba34f03a6152b0fbc57b4cb
I20260620 00:12:08.578588 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46126
I20260620 00:12:08.578595 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40634
I20260620 00:12:08.578680 2037 tablet_replica.cc:330] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: stopping tablet replica
I20260620 00:12:08.578688 2036 tablet_replica.cc:330] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: stopping tablet replica
I20260620 00:12:08.578727 2037 raft_consensus.cc:2229] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:08.578737 2036 raft_consensus.cc:2229] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:08.578761 2037 raft_consensus.cc:2258] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.578769 2036 raft_consensus.cc:2258] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.578918 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 816003de0ba34f03a6152b0fbc57b4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57408
I20260620 00:12:08.579006 2038 tablet_replica.cc:330] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: stopping tablet replica
I20260620 00:12:08.579057 2038 raft_consensus.cc:2229] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:08.579119 2038 raft_consensus.cc:2258] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:08.579106 2036 ts_tablet_manager.cc:1905] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.579190 2037 ts_tablet_manager.cc:1905] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.579416 2038 ts_tablet_manager.cc:1905] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:08.580920 2037 ts_tablet_manager.cc:1918] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:08.580952 2038 ts_tablet_manager.cc:1918] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:08.580989 2037 log.cc:1196] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/816003de0ba34f03a6152b0fbc57b4cb
I20260620 00:12:08.580996 2038 log.cc:1196] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/816003de0ba34f03a6152b0fbc57b4cb
I20260620 00:12:08.580998 2036 ts_tablet_manager.cc:1918] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:08.581049 2036 log.cc:1196] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/816003de0ba34f03a6152b0fbc57b4cb
I20260620 00:12:08.581224 2037 ts_tablet_manager.cc:1939] T 816003de0ba34f03a6152b0fbc57b4cb P cef4b6c6f6e14b29bef529af373f2edc: Deleting consensus metadata
I20260620 00:12:08.581223 2038 ts_tablet_manager.cc:1939] T 816003de0ba34f03a6152b0fbc57b4cb P 29c4589c8ebd4f92adb8d04df3a0a3ba: Deleting consensus metadata
I20260620 00:12:08.581295 2036 ts_tablet_manager.cc:1939] T 816003de0ba34f03a6152b0fbc57b4cb P 382cc7569c504d4f9225c11b79130355: Deleting consensus metadata
I20260620 00:12:08.581622 1491 catalog_manager.cc:4841] TS cef4b6c6f6e14b29bef529af373f2edc (127.0.98.2:38275): tablet 816003de0ba34f03a6152b0fbc57b4cb (table test-table [id=8f2ba5faa06845009b733696b8387f02]) successfully deleted
I20260620 00:12:08.581658 1492 catalog_manager.cc:4841] TS 382cc7569c504d4f9225c11b79130355 (127.0.98.3:35103): tablet 816003de0ba34f03a6152b0fbc57b4cb (table test-table [id=8f2ba5faa06845009b733696b8387f02]) successfully deleted
I20260620 00:12:08.581735 1492 catalog_manager.cc:4841] TS 29c4589c8ebd4f92adb8d04df3a0a3ba (127.0.98.1:36351): tablet 816003de0ba34f03a6152b0fbc57b4cb (table test-table [id=8f2ba5faa06845009b733696b8387f02]) successfully deleted
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914328, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:08.593781 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.581359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.0.98.1:36351)
0620 00:12:08.581404 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.581406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.584770 (+ 3364us) server_negotiation.cc:407] Connection header received
0620 00:12:08.584801 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.584802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.584823 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.584894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.585428 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.586185 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.586913 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.587212 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.587287 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.590521 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.590536 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.590536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.590545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.592523 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.592933 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.592937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.592937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.593005 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.593198 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.593199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.593199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.593383 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.593521 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.593590 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.593678 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:12:08.594419 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52904
I20260620 00:12:08.602063 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:08.595515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.0.98.62:34373)
0620 00:12:08.595584 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:08.595586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:08.595707 (+ 121us) server_negotiation.cc:407] Connection header received
0620 00:12:08.595784 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:08.595785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:08.595795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:08.595867 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:08.596407 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.597131 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.597892 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:08.598103 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:08.598188 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:08.599423 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:08.599424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:08.599425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:08.599432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:08.600931 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.601280 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.601282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.601282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.601345 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:08.601555 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:08.601556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:08.601556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:08.601673 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:08.601772 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:08.601877 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 00:12:08.601952 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 00:12:08.602309 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43582
I20260620 00:12:08.602887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1579
I20260620 00:12:08.609477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1719
I20260620 00:12:08.615473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1859
I20260620 00:12:08.620436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 1475
2026-06-20T00:12:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2524 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[2076](info): set up 2 sockets
Jun 20 00:12:08 dist-test-slave-28bv krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260620 00:12:09.672959 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:09 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:09.800031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:09.883059 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:09.883302 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:09.883325 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:09.884590 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:09.884640 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:12:09.884653 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:09.884667 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:09.884680 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:09.884692 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:09.886261 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:09.886541 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:09.886785 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:09.891942 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:09.891986 2089 server_base.cc:1034] running on GCE node
W20260620 00:12:09.893476 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:09.893503 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:09.893944 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:09.894286 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:09.895450 2089 hybrid_clock.cc:648] HybridClock initialized: now 1781914329895426 us; error 43 us; skew 500 ppm
Jun 20 00:12:09 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:09.898293 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:09.898780 2089 webserver.cc:460] Webserver started at http://127.0.98.62:39777/ using document root <none> and password file <none>
I20260620 00:12:09.899024 2089 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:09.899096 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:09.899255 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:09.900400 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7e3ff66409834a3793d6f3fdb16b41df"
format_stamp: "Formatted at 2026-06-20 00:12:09 on dist-test-slave-28bv"
server_key: "33e721fbbda32ff14a34b1da57e57005"
server_key_iv: "cb4e1ffead7d588789ec412d325e9d27"
server_key_version: "encryptionkey@0"
I20260620 00:12:09.900846 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e3ff66409834a3793d6f3fdb16b41df"
format_stamp: "Formatted at 2026-06-20 00:12:09 on dist-test-slave-28bv"
server_key: "33e721fbbda32ff14a34b1da57e57005"
server_key_iv: "cb4e1ffead7d588789ec412d325e9d27"
server_key_version: "encryptionkey@0"
I20260620 00:12:09.902246 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:09.903084 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:09.903268 2089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:09.903383 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7e3ff66409834a3793d6f3fdb16b41df"
format_stamp: "Formatted at 2026-06-20 00:12:09 on dist-test-slave-28bv"
server_key: "33e721fbbda32ff14a34b1da57e57005"
server_key_iv: "cb4e1ffead7d588789ec412d325e9d27"
server_key_version: "encryptionkey@0"
I20260620 00:12:09.903461 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:09.915320 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:09.917804 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:09.917984 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:09.922626 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44529
I20260620 00:12:09.922755 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44529 every 8 connection(s)
I20260620 00:12:09.923065 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:09.923741 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:09.925043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2089
I20260620 00:12:09.925148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:09.925343 392 external_mini_cluster.cc:1428] Setting key 19cd0bd1978905db601e9bf07dcf5a2f
I20260620 00:12:09.926506 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Bootstrap starting.
I20260620 00:12:09.927281 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:09.927632 2167 log.cc:824] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:09.928385 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: No bootstrap required, opened a new log
I20260620 00:12:09.930752 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260620 00:12:09.930907 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:09.930930 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3ff66409834a3793d6f3fdb16b41df, State: Initialized, Role: FOLLOWER
I20260620 00:12:09.931092 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260620 00:12:09.931185 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:09.931226 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:09.931268 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:09.932116 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260620 00:12:09.932258 2167 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3ff66409834a3793d6f3fdb16b41df; no voters:
I20260620 00:12:09.932478 2167 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:09.932502 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:09.932637 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [term 1 LEADER]: Becoming Leader. State: Replica: 7e3ff66409834a3793d6f3fdb16b41df, State: Running, Role: LEADER
I20260620 00:12:09.932734 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } }
I20260620 00:12:09.932789 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 00:12:09 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:09.933070 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3ff66409834a3793d6f3fdb16b41df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } } }
I20260620 00:12:09.933143 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:09.933280 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3ff66409834a3793d6f3fdb16b41df. Latest consensus state: current_term: 1 leader_uuid: "7e3ff66409834a3793d6f3fdb16b41df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ff66409834a3793d6f3fdb16b41df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44529 } } }
I20260620 00:12:09.933350 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:09.933467 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:09.934103 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:09.935997 2177 catalog_manager.cc:1347] Generated new cluster ID: 1513c2f71ae74ed79352841f4a12a6b9
I20260620 00:12:09.936054 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:09.937860 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:09.925973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.0.98.62:44529)
0620 00:12:09.926157 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:09.926166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:09.926191 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:12:09.926591 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:09.926598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:09.926688 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:09.926821 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:09.927681 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:09.928780 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:09.929927 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:09.930354 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:09.930441 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:09.933662 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:09.933667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:09.933667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:09.933681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:09.936046 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:09.936499 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:09.936502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:09.936503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:09.936639 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:09.936875 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:09.936876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:09.936876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:09.937195 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:09.937340 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:09.937471 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 00:12:09.937629 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260620 00:12:09.950346 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:09.950923 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:09.955927 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Generated new TSK 0
I20260620 00:12:09.956161 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:09.993309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:10.081091 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:10.081355 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:10.081401 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:10.082640 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:10.082713 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:10.082751 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:12:10.082783 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:10.082816 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:10.084362 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:10.084673 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:10.084928 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:10.090334 2193 server_base.cc:1034] running on GCE node
W20260620 00:12:10.090291 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.090306 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.090306 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:10.090791 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:10.091064 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:10.092272 2193 hybrid_clock.cc:648] HybridClock initialized: now 1781914330092191 us; error 78 us; skew 500 ppm
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:10.095023 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:10.095600 2193 webserver.cc:460] Webserver started at http://127.0.98.1:45835/ using document root <none> and password file <none>
I20260620 00:12:10.095857 2193 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:10.095928 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:10.096079 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:10.097234 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc506c17ca474ec8a9e69402bda31131"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "84044dcbf9436a5ad0ba933750b271c3"
server_key_iv: "b458fbfa42deadf209fb0ed559630cfb"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.097632 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc506c17ca474ec8a9e69402bda31131"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "84044dcbf9436a5ad0ba933750b271c3"
server_key_iv: "b458fbfa42deadf209fb0ed559630cfb"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.099030 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:10.099843 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.100042 2193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:10.100150 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "cc506c17ca474ec8a9e69402bda31131"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "84044dcbf9436a5ad0ba933750b271c3"
server_key_iv: "b458fbfa42deadf209fb0ed559630cfb"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.100265 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:10.118011 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:10.120499 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:10.120707 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:10.121340 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:10.121393 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.121425 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:10.121464 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:10.129115 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37617
I20260620 00:12:10.129206 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37617 every 8 connection(s)
I20260620 00:12:10.129501 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:10.131806 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.121914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40125 (local address 127.0.98.62:44529)
0620 00:12:10.121991 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.121994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.122193 (+ 199us) server_negotiation.cc:407] Connection header received
0620 00:12:10.122687 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.122688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.122700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.122791 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.123433 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.124283 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.125519 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.125897 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.125984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.128667 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.128669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.128669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.128678 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.130374 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.130800 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.130801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.130802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.130869 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.131125 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.131126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.131127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.131258 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.131337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.131608 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.131709 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:12:10.131996 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.121881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.1:40125)
0620 00:12:10.122051 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.122093 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.122592 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.122855 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.122859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.123161 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.123350 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.123355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.124456 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.124457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.125409 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.125411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.126010 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.126011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.126175 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.126757 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.126773 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.128527 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.130478 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.130485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.130487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.130719 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.130938 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.130939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.130940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.131050 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.131328 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.131334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.131474 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.131587 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.131760 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260620 00:12:10.132658 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260620 00:12:10.132735 2324 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:10.132928 2324 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
I20260620 00:12:10.133498 2120 ts_manager.cc:194] Registered new tserver with Master: cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617)
I20260620 00:12:10.134294 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40125
I20260620 00:12:10.139155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2193
I20260620 00:12:10.139400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:10.139593 392 external_mini_cluster.cc:1428] Setting key ae2e67e1d3694070fa90b91d7a985be9
I20260620 00:12:10.140196 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.133952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60435 (local address 127.0.98.62:44529)
0620 00:12:10.133998 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.134000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.134128 (+ 128us) server_negotiation.cc:407] Connection header received
0620 00:12:10.134168 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.134169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.134180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.134237 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.134612 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.135312 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.136132 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.136338 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.136403 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.137460 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.137461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.137461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.137467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.138864 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.139344 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.139345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.139346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.139399 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.139610 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.139611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.139611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.139772 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.139858 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.140026 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.140105 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:12:10.140221 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.133898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.1:60435)
0620 00:12:10.134053 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.134065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.134121 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.134239 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.134239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.134433 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.134515 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.134516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.135500 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.135501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.136032 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.136034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.136418 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.136419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.136484 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.136979 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.136984 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.137376 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.138963 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.138964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.138965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.139133 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.139479 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.139479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.139480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.139543 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.139849 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.139850 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.139901 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.140001 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.140106 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:10.171802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:10.258154 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:10.258419 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:10.258461 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:10.259677 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:10.259752 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:10.259788 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:12:10.259819 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:10.259859 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:10.261409 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:10.261698 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:10.261936 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:10.267258 2333 server_base.cc:1034] running on GCE node
W20260620 00:12:10.267225 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.267274 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.267225 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:10.267697 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:10.267990 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:10.269191 2333 hybrid_clock.cc:648] HybridClock initialized: now 1781914330269157 us; error 55 us; skew 500 ppm
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:10.272009 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:10.272758 2333 webserver.cc:460] Webserver started at http://127.0.98.2:40713/ using document root <none> and password file <none>
I20260620 00:12:10.273065 2333 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:10.273145 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:10.273286 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:10.274595 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e8f735199d44431b1999081a6e8bff0"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "5eb0f03d0eef8bc19fb05352c5904979"
server_key_iv: "8bc9505cda164aaac8f26dff91aec44e"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.274977 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e8f735199d44431b1999081a6e8bff0"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "5eb0f03d0eef8bc19fb05352c5904979"
server_key_iv: "8bc9505cda164aaac8f26dff91aec44e"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.276383 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:10.277352 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.277612 2333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:10.277741 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "3e8f735199d44431b1999081a6e8bff0"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "5eb0f03d0eef8bc19fb05352c5904979"
server_key_iv: "8bc9505cda164aaac8f26dff91aec44e"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.277835 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:10.289256 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:10.291442 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:10.291630 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:10.292220 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:10.292281 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.292322 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:10.292343 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:10.299998 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39875
I20260620 00:12:10.300124 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39875 every 8 connection(s)
I20260620 00:12:10.300400 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:10.302644 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.292691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55325 (local address 127.0.98.62:44529)
0620 00:12:10.292777 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.292780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.293092 (+ 312us) server_negotiation.cc:407] Connection header received
0620 00:12:10.293883 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.293884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.293897 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.293966 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.294966 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.295832 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.297069 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.297276 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.297312 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.299613 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.299614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.299615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.299623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.301179 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.301657 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.301658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.301659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.301716 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.301973 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.301974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.301975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.302098 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.302177 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.302465 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.302546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:12:10.302848 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.292691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.2:55325)
0620 00:12:10.292927 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.292983 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.293723 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.294080 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.294086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.294518 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.294835 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.294843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.295985 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.295986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.296938 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.296940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.297400 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.297401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.297499 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.298050 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.298061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.299509 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.301295 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.301302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.301305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.301564 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.301794 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.301795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.301796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.301906 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.302181 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.302188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.302336 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.302446 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.302613 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":128,"threads_started":1}
I20260620 00:12:10.303356 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260620 00:12:10.303440 2464 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:10.303622 2464 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
I20260620 00:12:10.303999 2121 ts_manager.cc:194] Registered new tserver with Master: 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875)
I20260620 00:12:10.304457 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55325
I20260620 00:12:10.307399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2333
I20260620 00:12:10.307499 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:10.307704 392 external_mini_cluster.cc:1428] Setting key 749ada1724c5a1ebb59a7978efba6353
I20260620 00:12:10.310747 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.304477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39913 (local address 127.0.98.62:44529)
0620 00:12:10.304517 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.304519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.304673 (+ 154us) server_negotiation.cc:407] Connection header received
0620 00:12:10.304743 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.304744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.304755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.304803 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.305114 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.305592 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.306350 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.306574 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.306640 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.307826 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.307827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.307828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.307835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.309423 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.309922 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.309923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.309924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.309984 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.310227 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.310228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.310228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.310379 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.310438 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.310613 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.310684 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:10.310866 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.304430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.2:39913)
0620 00:12:10.304596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.304607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.304687 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.304805 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.304806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.304984 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.305039 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.305040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.305688 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.305689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.306253 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.306254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.306670 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.306671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.306739 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.307257 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.307263 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.307692 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.309568 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.309570 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.309570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.309823 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.310060 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.310061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.310061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.310133 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.310447 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.310448 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.310509 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.310583 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.310706 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:10.336735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44529
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:10.425209 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:10.425454 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:10.425498 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:10.426720 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:10.426800 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:10.426837 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:12:10.426868 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:12:10.426903 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:10.428660 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:10.428946 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:10.429186 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:10.435055 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.435065 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:10.435072 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:10.435473 2473 server_base.cc:1034] running on GCE node
I20260620 00:12:10.435747 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:10.436024 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:10.437220 2473 hybrid_clock.cc:648] HybridClock initialized: now 1781914330437213 us; error 49 us; skew 500 ppm
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:10.440212 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:10.440737 2473 webserver.cc:460] Webserver started at http://127.0.98.3:34017/ using document root <none> and password file <none>
I20260620 00:12:10.440994 2473 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:10.441040 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:10.441180 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:10.442278 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "896ecc145ee74d508be723558d569349"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "3913db2ee4c1a9711aa7093e5239f3fb"
server_key_iv: "57a09a95824b138a361d11f7160cc971"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.442693 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "896ecc145ee74d508be723558d569349"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "3913db2ee4c1a9711aa7093e5239f3fb"
server_key_iv: "57a09a95824b138a361d11f7160cc971"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.444092 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:10.445045 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.445313 2473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:10.445417 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "896ecc145ee74d508be723558d569349"
format_stamp: "Formatted at 2026-06-20 00:12:10 on dist-test-slave-28bv"
server_key: "3913db2ee4c1a9711aa7093e5239f3fb"
server_key_iv: "57a09a95824b138a361d11f7160cc971"
server_key_version: "encryptionkey@0"
I20260620 00:12:10.445518 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:10.471740 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:10.474391 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:10.474519 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:10.475104 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:10.475172 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.475220 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:10.475245 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:10.482837 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46283
I20260620 00:12:10.482942 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46283 every 8 connection(s)
I20260620 00:12:10.483250 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:10.485553 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.475693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36977 (local address 127.0.98.62:44529)
0620 00:12:10.475740 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.475743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.476016 (+ 273us) server_negotiation.cc:407] Connection header received
0620 00:12:10.476656 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.476658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.476674 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.476752 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.477446 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.478309 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.479473 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.479697 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.479792 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.482349 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.482351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.482352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.482361 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.484009 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.484483 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.484485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.484486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.484592 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.484867 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.484868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.484868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.485008 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.485080 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.485375 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.485461 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:12:10.485766 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.475697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.3:36977)
0620 00:12:10.475889 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.475926 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.476483 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.476875 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.476880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.477156 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.477368 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.477372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.478471 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.478473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.479369 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.479371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.479816 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.479817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.479988 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.480663 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.480673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.482233 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.484116 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.484124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.484126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.484399 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.484673 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.484674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.484674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.484793 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.485085 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.485091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.485233 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.485343 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.485512 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":109,"threads_started":1}
I20260620 00:12:10.486444 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44529
I20260620 00:12:10.486541 2604 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:10.486732 2604 heartbeater.cc:511] Master 127.0.98.62:44529 requested a full tablet report, sending...
I20260620 00:12:10.487182 2121 ts_manager.cc:194] Registered new tserver with Master: 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:10.487741 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36977
I20260620 00:12:10.492619 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.487507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50925 (local address 127.0.98.62:44529)
0620 00:12:10.487546 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.487547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.487645 (+ 98us) server_negotiation.cc:407] Connection header received
0620 00:12:10.487703 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.487703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.487710 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.487758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.488148 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.488676 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.489362 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.489538 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.489590 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.490564 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.490565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.490566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.490571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.491689 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.491990 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.491991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.491992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.492038 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.492181 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.492182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.492182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.492275 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.492324 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.492438 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.492494 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:12:10.492677 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.487454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.98.3:50925)
0620 00:12:10.487588 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.487599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.487650 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.487763 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.487764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.487965 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.488077 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.488079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.488762 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.488763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.489260 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.489262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.489603 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.489603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.489657 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.490136 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.490141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.490487 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.491765 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.491766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.491766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.491926 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.492078 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.492078 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.492078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.492133 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.492331 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.492332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.492368 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.492420 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.492507 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 00:12:10.493300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2473
I20260620 00:12:10.493396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:10.493599 392 external_mini_cluster.cc:1428] Setting key 1339f104ceeb835b308d23147813d9d1
I20260620 00:12:10.494421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:10.519016 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.511275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.0.98.62:44529)
0620 00:12:10.511343 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.511345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.511354 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:10.511379 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.511380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.511389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.511456 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.512079 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.512694 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.513496 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.513696 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.513782 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.516264 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.516266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.516266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.516278 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.517763 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.518172 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.518173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.518173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.518233 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.518450 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.518451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.518451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.518570 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.518674 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.518824 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.518910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:12:10.521219 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:10.522876 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:10.529968 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.525748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39875 (local address 127.0.0.1:41046)
0620 00:12:10.526082 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.526095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.526173 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.526517 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.526520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.526522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.526616 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.526618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.527533 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.527534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.528825 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.528827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.529603 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.529689 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.529710 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.529808 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.529866 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260620 00:12:10.529976 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.525898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.2:39875)
0620 00:12:10.526175 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.526179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.526190 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:10.526338 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.526341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.526358 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.526419 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.526693 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.527442 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.528914 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.529489 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529599 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.529670 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.529827 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.529866 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":84,"threads_started":1}
I20260620 00:12:10.530325 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.525948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46283 (local address 127.0.0.1:46794)
0620 00:12:10.526358 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.526368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.526429 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.526602 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.526602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.526604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.526642 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.526643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.528014 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.528015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.529137 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.529139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.530096 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.530096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.530166 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.530170 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.530221 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.530257 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":38,"thread_start_us":65,"threads_started":1}
I20260620 00:12:10.530568 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.526140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.0.98.3:46283)
0620 00:12:10.526309 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.526313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.526442 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:12:10.526491 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.526493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.526510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.526591 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.526765 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.527890 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529236 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.529949 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529989 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.530036 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.530261 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.530304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260620 00:12:10.530771 2398 tablet_service.cc:1467] Processing CreateTablet for tablet a9eff746f7be4c1293fad3c51780da84 (DEFAULT_TABLE table=test-table [id=1fb9f459607d408c92e09e4929917f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:10.530936 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.525748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37617 (local address 127.0.0.1:57856)
0620 00:12:10.525923 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.525964 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.526395 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.526680 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.526681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.526683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.526740 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.526742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.528002 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.528003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.528973 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.528975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.529959 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.530591 (+ 632us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.530597 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.530823 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.530872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260620 00:12:10.531008 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.525891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.0.98.1:37617)
0620 00:12:10.526098 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.526104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.526363 (+ 259us) server_negotiation.cc:407] Connection header received
0620 00:12:10.526515 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.526517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.526535 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.526588 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.526805 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.527911 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529059 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.529788 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.529860 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.529914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.530867 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.530911 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
I20260620 00:12:10.531180 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9eff746f7be4c1293fad3c51780da84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.531206 2538 tablet_service.cc:1467] Processing CreateTablet for tablet a9eff746f7be4c1293fad3c51780da84 (DEFAULT_TABLE table=test-table [id=1fb9f459607d408c92e09e4929917f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:10.531553 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9eff746f7be4c1293fad3c51780da84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.531903 2258 tablet_service.cc:1467] Processing CreateTablet for tablet a9eff746f7be4c1293fad3c51780da84 (DEFAULT_TABLE table=test-table [id=1fb9f459607d408c92e09e4929917f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:10.532251 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9eff746f7be4c1293fad3c51780da84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.534602 2627 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Bootstrap starting.
I20260620 00:12:10.535188 2627 tablet_bootstrap.cc:654] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.535421 2627 log.cc:824] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:10.535773 2630 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Bootstrap starting.
I20260620 00:12:10.536095 2627 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: No bootstrap required, opened a new log
I20260620 00:12:10.536150 2627 ts_tablet_manager.cc:1397] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:10.536077 2628 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Bootstrap starting.
I20260620 00:12:10.536365 2630 tablet_bootstrap.cc:654] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.536779 2630 log.cc:824] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:10.537271 2628 tablet_bootstrap.cc:654] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.537640 2628 log.cc:824] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:10.537712 2630 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: No bootstrap required, opened a new log
I20260620 00:12:10.537781 2630 ts_tablet_manager.cc:1397] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:10.537840 2627 raft_consensus.cc:348] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.538017 2627 raft_consensus.cc:374] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.538064 2627 raft_consensus.cc:729] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.538204 2627 consensus_queue.cc:260] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.538496 2627 ts_tablet_manager.cc:1428] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:12:10.538538 2464 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
I20260620 00:12:10.538964 2628 tablet_bootstrap.cc:492] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: No bootstrap required, opened a new log
I20260620 00:12:10.539067 2628 ts_tablet_manager.cc:1397] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:12:10.539815 2630 raft_consensus.cc:348] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.539978 2630 raft_consensus.cc:374] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.540014 2630 raft_consensus.cc:729] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.540123 2630 consensus_queue.cc:260] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.540380 2630 ts_tablet_manager.cc:1428] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:10.540421 2324 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
I20260620 00:12:10.541174 2628 raft_consensus.cc:348] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.541345 2628 raft_consensus.cc:374] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.541395 2628 raft_consensus.cc:729] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ecc145ee74d508be723558d569349, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.541507 2628 consensus_queue.cc:260] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.541782 2628 ts_tablet_manager.cc:1428] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:12:10.541795 2604 heartbeater.cc:503] Master 127.0.98.62:44529 was elected leader, sending a full tablet report...
W20260620 00:12:10.551002 2465 tablet.cc:2369] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:10.609056 2634 raft_consensus.cc:482] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:10.609285 2634 raft_consensus.cc:504] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.609725 2634 leader_election.cc:288] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:10.614025 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.610002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53617 (local address 127.0.98.2:39875)
0620 00:12:10.610046 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.610047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.610060 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:12:10.610097 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.610098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.610108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.610224 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.610454 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.611243 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.612987 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.613583 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.613672 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.613683 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.613912 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.613953 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:12:10.614025 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.609989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45543 (local address 127.0.98.3:46283)
0620 00:12:10.610036 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.610039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.610206 (+ 167us) server_negotiation.cc:407] Connection header received
0620 00:12:10.610236 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.610237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.610250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.610311 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.610452 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.611277 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.612986 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.613589 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.613672 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.613683 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.613912 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.613953 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:12:10.614039 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.609936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39875 (local address 127.0.98.1:53617)
0620 00:12:10.609976 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.609989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.610035 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.610257 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.610257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.610259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.610350 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.610353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.611388 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.611390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.612828 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.612832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.613712 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.613713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.613810 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.613817 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.613886 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.613941 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":84}
I20260620 00:12:10.614048 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.609936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46283 (local address 127.0.98.1:45543)
0620 00:12:10.610114 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.610127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.610178 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.610325 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.610326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.610327 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.610383 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.610384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.611388 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.611390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.612828 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.612832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.613712 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.613713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.613810 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.613817 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.613886 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.613941 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":178,"thread_start_us":88,"threads_started":1}
I20260620 00:12:10.614446 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9eff746f7be4c1293fad3c51780da84" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0" is_pre_election: true
I20260620 00:12:10.614487 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9eff746f7be4c1293fad3c51780da84" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349" is_pre_election: true
I20260620 00:12:10.614610 2418 raft_consensus.cc:2454] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:10.614638 2558 raft_consensus.cc:2454] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:10.614849 2211 leader_election.cc:302] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:10.614975 2634 raft_consensus.cc:2781] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:10.615031 2634 raft_consensus.cc:482] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:10.615054 2634 raft_consensus.cc:3037] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.615741 2634 raft_consensus.cc:504] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.615892 2634 leader_election.cc:288] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:10.616050 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9eff746f7be4c1293fad3c51780da84" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0"
I20260620 00:12:10.616051 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9eff746f7be4c1293fad3c51780da84" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349"
I20260620 00:12:10.616132 2558 raft_consensus.cc:3037] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.616132 2418 raft_consensus.cc:3037] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.617089 2558 raft_consensus.cc:2454] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:10.617089 2418 raft_consensus.cc:2454] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:10.617321 2211 leader_election.cc:302] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:10.617419 2634 raft_consensus.cc:2781] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:10.617563 2634 raft_consensus.cc:686] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Becoming Leader. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Running, Role: LEADER
I20260620 00:12:10.617658 2634 consensus_queue.cc:237] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.618544 2121 catalog_manager.cc:5495] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc506c17ca474ec8a9e69402bda31131 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc506c17ca474ec8a9e69402bda31131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } health_report { overall_health: UNKNOWN } } }
W20260620 00:12:10.630259 2325 tablet.cc:2369] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:12:10 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:10.648835 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.641278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44529 (local address 127.0.0.1:59260)
0620 00:12:10.641513 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.641525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.641647 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.641795 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.641796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.642121 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:10.642172 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.642175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.642843 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.642844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.643365 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.643366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.643728 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.643729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.643779 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.644378 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:10.644382 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:10.645915 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:10.647611 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.647617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.647619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.647826 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.648050 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:10.648051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:10.648051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:10.648106 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:10.648393 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:10.648400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:10.648449 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.648529 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.648737 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":138,"threads_started":1}
I20260620 00:12:10.648912 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.641356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.0.98.62:44529)
0620 00:12:10.641466 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.641468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.641625 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:12:10.641710 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.641711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.641721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.641777 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:10.642262 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.642753 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.643450 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.643642 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.643728 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.646020 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:10.646022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:10.646022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:10.646029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:10.647504 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.647926 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.647927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.647927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.647983 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:10.648164 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:10.648164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:10.648165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:10.648272 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:10.648375 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.648660 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.648788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":50}
I20260620 00:12:10.651314 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:12:10.651551 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:10.653236 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f7156954272902d908d21198197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecea825e17ee4239ac559d7c5bfc7265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:10.653225 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f7156954272902d908d21198197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecea825e17ee4239ac559d7c5bfc7265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:10.653277 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 5f953f7156954272902d908d21198197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecea825e17ee4239ac559d7c5bfc7265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:12:10.653398 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f953f7156954272902d908d21198197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.653396 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f953f7156954272902d908d21198197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.653446 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f953f7156954272902d908d21198197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:10.654677 2627 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: Bootstrap starting.
I20260620 00:12:10.654934 2628 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: Bootstrap starting.
I20260620 00:12:10.655455 2627 tablet_bootstrap.cc:654] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.655503 2628 tablet_bootstrap.cc:654] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.655849 2630 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: Bootstrap starting.
I20260620 00:12:10.656231 2628 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: No bootstrap required, opened a new log
I20260620 00:12:10.656234 2630 tablet_bootstrap.cc:654] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:10.656291 2628 ts_tablet_manager.cc:1397] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:10.656441 2627 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: No bootstrap required, opened a new log
I20260620 00:12:10.656489 2627 ts_tablet_manager.cc:1397] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:10.656455 2628 raft_consensus.cc:348] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.656580 2628 raft_consensus.cc:374] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.656615 2628 raft_consensus.cc:729] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ecc145ee74d508be723558d569349, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.656654 2628 consensus_queue.cc:260] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.656661 2627 raft_consensus.cc:348] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.656725 2627 raft_consensus.cc:374] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.656752 2627 raft_consensus.cc:729] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.656765 2628 ts_tablet_manager.cc:1428] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:10.656792 2627 consensus_queue.cc:260] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.656880 2627 ts_tablet_manager.cc:1428] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.656854 2635 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:10.656949 2635 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:10.656936 2633 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:10.657011 2633 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:10.657467 2630 tablet_bootstrap.cc:492] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: No bootstrap required, opened a new log
I20260620 00:12:10.657531 2630 ts_tablet_manager.cc:1397] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:10.657677 2630 raft_consensus.cc:348] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.657752 2630 raft_consensus.cc:374] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:10.657775 2630 raft_consensus.cc:729] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Initialized, Role: FOLLOWER
I20260620 00:12:10.657823 2630 consensus_queue.cc:260] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.657909 2630 ts_tablet_manager.cc:1428] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:10.657965 2634 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:10.658077 2634 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 00:12:10.734540 2605 tablet.cc:2369] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:10.976117 2633 raft_consensus.cc:482] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:10.976234 2633 raft_consensus.cc:504] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.976577 2633 leader_election.cc:288] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:10.979895 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.976918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41731 (local address 127.0.98.1:37617)
0620 00:12:10.977016 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.977019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.977108 (+ 89us) server_negotiation.cc:407] Connection header received
0620 00:12:10.977228 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.977233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.977244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.977314 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.977493 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.977987 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979131 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.979503 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979603 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.979615 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.979773 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.979809 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 00:12:10.979928 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.977170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45767 (local address 127.0.98.3:46283)
0620 00:12:10.977220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:10.977222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:10.977251 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:12:10.977278 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:10.977279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:10.977289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:10.977364 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:10.977522 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.977981 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979153 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.979527 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979635 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.979647 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:10.979816 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 00:12:10.979859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:12:10.979950 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.977045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46283 (local address 127.0.98.2:45767)
0620 00:12:10.977143 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.977154 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.977208 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.977367 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.977368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.977370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.977432 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.977434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.978076 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.978077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.979009 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.979020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.979658 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.979734 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.979737 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.979799 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.979840 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":37,"threads_started":1}
I20260620 00:12:10.979967 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:10.976852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37617 (local address 127.0.98.2:41731)
0620 00:12:10.977010 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:10.977023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:10.977141 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:10.977326 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:10.977327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:10.977329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:10.977407 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.977410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.978085 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.978086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.979012 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:10.979014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:10.979605 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:10.979606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:10.979681 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:10.979685 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:10.979767 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:12:10.979809 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":141,"thread_start_us":76,"threads_started":1}
I20260620 00:12:10.980321 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f7156954272902d908d21198197" candidate_uuid: "3e8f735199d44431b1999081a6e8bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc506c17ca474ec8a9e69402bda31131" is_pre_election: true
I20260620 00:12:10.980425 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f7156954272902d908d21198197" candidate_uuid: "3e8f735199d44431b1999081a6e8bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349" is_pre_election: true
I20260620 00:12:10.980468 2278 raft_consensus.cc:2454] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8f735199d44431b1999081a6e8bff0 in term 0.
I20260620 00:12:10.980541 2558 raft_consensus.cc:2454] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e8f735199d44431b1999081a6e8bff0 in term 0.
I20260620 00:12:10.980726 2350 leader_election.cc:302] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:10.980890 2633 raft_consensus.cc:2781] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:10.980933 2633 raft_consensus.cc:482] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:10.980953 2633 raft_consensus.cc:3037] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.981805 2633 raft_consensus.cc:504] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.981963 2633 leader_election.cc:288] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [CANDIDATE]: Term 1 election: Requested vote from peers cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:10.982180 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f7156954272902d908d21198197" candidate_uuid: "3e8f735199d44431b1999081a6e8bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc506c17ca474ec8a9e69402bda31131"
I20260620 00:12:10.982185 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f953f7156954272902d908d21198197" candidate_uuid: "3e8f735199d44431b1999081a6e8bff0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349"
I20260620 00:12:10.982271 2278 raft_consensus.cc:3037] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.982271 2558 raft_consensus.cc:3037] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:10.983062 2278 raft_consensus.cc:2454] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8f735199d44431b1999081a6e8bff0 in term 1.
I20260620 00:12:10.983062 2558 raft_consensus.cc:2454] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e8f735199d44431b1999081a6e8bff0 in term 1.
I20260620 00:12:10.983245 2350 leader_election.cc:302] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:10.983372 2633 raft_consensus.cc:2781] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:10.983516 2633 raft_consensus.cc:686] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [term 1 LEADER]: Becoming Leader. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Running, Role: LEADER
I20260620 00:12:10.983628 2633 consensus_queue.cc:237] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:10.984095 2650 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: TxnStatusTablet state changed. Reason: New leader 3e8f735199d44431b1999081a6e8bff0. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:10.984185 2650 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:10.984355 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:12:10.984459 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:12:10.984524 2120 catalog_manager.cc:5495] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:11.013813 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.010850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.2:39875)
0620 00:12:11.010913 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:11.010914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:11.010923 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:11.010962 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:11.010963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:11.011003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:11.011075 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:11.011212 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.011729 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.012896 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.013111 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.013180 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.013423 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.013526 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.013601 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:11.013723 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 00:12:11.013749 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:12:11.013823 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.010740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39875 (local address 127.0.0.1:41056)
0620 00:12:11.010821 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:11.010832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:11.010874 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:11.011094 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:11.011094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:11.011096 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:11.011150 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.011152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.011814 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.011815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.012763 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.012766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.013199 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.013200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.013271 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.013292 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.013607 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.013617 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:12:11.013674 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:12:11.013731 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:12:11.015925 2278 raft_consensus.cc:1261] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Refusing update from remote peer 3e8f735199d44431b1999081a6e8bff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.015935 2558 raft_consensus.cc:1261] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Refusing update from remote peer 3e8f735199d44431b1999081a6e8bff0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.016286 2633 consensus_queue.cc:1035] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:11.016455 2650 consensus_queue.cc:1035] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:11.017551 2635 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: TxnStatusTablet state changed. Reason: New leader 3e8f735199d44431b1999081a6e8bff0. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.017643 2635 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:11.017593 2634 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: TxnStatusTablet state changed. Reason: New leader 3e8f735199d44431b1999081a6e8bff0. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.017684 2634 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:11.019140 2650 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.019239 2650 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:11.019254 2656 mvcc.cc:205] Tried to move back new op lower bound from 7298721099838525440 to 7298721099709759488. Current Snapshot: MvccSnapshot[applied={T|T < 7298721099838525440}]
I20260620 00:12:11.019284 2650 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.019321 2650 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 3e8f735199d44431b1999081a6e8bff0: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:12:11.019495 2635 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.019578 2635 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P 896ecc145ee74d508be723558d569349: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:11.020404 2634 tablet_replica.cc:439] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e8f735199d44431b1999081a6e8bff0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } } }
I20260620 00:12:11.020498 2634 tablet_replica.cc:442] T 5f953f7156954272902d908d21198197 P cc506c17ca474ec8a9e69402bda31131: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:12:11.021219 2657 mvcc.cc:205] Tried to move back new op lower bound from 7298721099838525440 to 7298721099709759488. Current Snapshot: MvccSnapshot[applied={T|T < 7298721099838525440 or (T in {7298721099838525440})}]
I20260620 00:12:11.024818 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.021778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.98.1:37617)
0620 00:12:11.021838 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:11.021840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:11.021969 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:12:11.022040 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:11.022040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:11.022055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:11.022114 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:11.022249 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.022709 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.023901 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.024128 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.024193 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.024406 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.024588 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.024686 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:11.024710 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 00:12:11.024736 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:12:11.030148 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.027413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42687 (local address 127.0.98.2:39875)
0620 00:12:11.027452 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:11.027454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:11.027503 (+ 49us) server_negotiation.cc:407] Connection header received
0620 00:12:11.027575 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:11.027575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:11.027594 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:11.027647 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:11.027779 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.028303 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.029467 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.029651 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.029721 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.029928 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.029955 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.029994 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:11.030072 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:12:11.030088 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:12:11.030234 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.027288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39875 (local address 127.0.98.1:42687)
0620 00:12:11.027449 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:11.027459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:11.027517 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:11.027652 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:11.027653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:11.027654 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:11.027720 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.027721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.028410 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.028411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.029370 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.029373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.029738 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.029739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.029804 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.029821 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.030015 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.030017 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:12:11.030070 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:12:11.030119 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260620 00:12:11.033078 2558 raft_consensus.cc:1261] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.033442 2418 raft_consensus.cc:1261] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.033387 2637 consensus_queue.cc:1035] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:11.033769 2637 consensus_queue.cc:1035] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:11.047120 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.043983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42317 (local address 127.0.98.1:37617)
0620 00:12:11.044022 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:11.044024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:11.044068 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:12:11.044091 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:11.044092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:11.044103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:11.044148 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:11.044279 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.044852 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.046145 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.046518 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.046565 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.046846 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.046882 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.046955 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:11.047016 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:12:11.047037 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:12:11.047140 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:11.043779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37617 (local address 127.0.98.2:42317)
0620 00:12:11.043961 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:11.043973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:11.044025 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:11.044157 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:11.044157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:11.044159 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:12:11.044228 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.044230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.044967 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.044968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.046043 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:11.046046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:11.046649 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:11.046650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:11.046735 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:11.046750 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:12:11.046956 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:12:11.046959 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:11.047012 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:12:11.047055 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
I20260620 00:12:11.063118 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:11.063251 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:11.064077 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Sending DeleteTablet for 3 replicas of tablet a9eff746f7be4c1293fad3c51780da84
I20260620 00:12:11.064466 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet a9eff746f7be4c1293fad3c51780da84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41046
I20260620 00:12:11.064498 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet a9eff746f7be4c1293fad3c51780da84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57856
I20260620 00:12:11.064728 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet a9eff746f7be4c1293fad3c51780da84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46794
I20260620 00:12:11.064910 2678 tablet_replica.cc:330] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: stopping tablet replica
I20260620 00:12:11.065064 2678 raft_consensus.cc:2229] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:11.065183 2678 raft_consensus.cc:2258] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:11.065188 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:11.065337 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:11.065876 2678 ts_tablet_manager.cc:1905] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:11.066943 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (DEFAULT_TABLE table=test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:11.067092 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (DEFAULT_TABLE table=test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:11.067209 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1356ea92245ecaaa4b9e86d38a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:11.067374 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1356ea92245ecaaa4b9e86d38a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:11.068671 2628 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Bootstrap starting.
I20260620 00:12:11.068683 2627 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Bootstrap starting.
I20260620 00:12:11.069058 2628 tablet_bootstrap.cc:654] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:11.069106 2627 tablet_bootstrap.cc:654] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:11.066897 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (DEFAULT_TABLE table=test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:11.070180 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1356ea92245ecaaa4b9e86d38a0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:11.070492 2678 ts_tablet_manager.cc:1918] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:11.070605 2678 log.cc:1196] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/a9eff746f7be4c1293fad3c51780da84
I20260620 00:12:11.070940 2678 ts_tablet_manager.cc:1939] T a9eff746f7be4c1293fad3c51780da84 P 896ecc145ee74d508be723558d569349: Deleting consensus metadata
I20260620 00:12:11.071532 2630 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Bootstrap starting.
I20260620 00:12:11.071614 2106 catalog_manager.cc:4841] TS 896ecc145ee74d508be723558d569349 (127.0.98.3:46283): tablet a9eff746f7be4c1293fad3c51780da84 (table test-table [id=1fb9f459607d408c92e09e4929917f2c]) successfully deleted
I20260620 00:12:11.071954 2628 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: No bootstrap required, opened a new log
I20260620 00:12:11.071962 2630 tablet_bootstrap.cc:654] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:11.071997 2628 ts_tablet_manager.cc:1397] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:12:11.072160 2628 raft_consensus.cc:348] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.072234 2628 raft_consensus.cc:374] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:11.072254 2628 raft_consensus.cc:729] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ecc145ee74d508be723558d569349, State: Initialized, Role: FOLLOWER
I20260620 00:12:11.072297 2628 consensus_queue.cc:260] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.072379 2628 ts_tablet_manager.cc:1428] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:11.065618 2676 tablet_replica.cc:330] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: stopping tablet replica
I20260620 00:12:11.072803 2676 raft_consensus.cc:2229] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:11.072918 2676 raft_consensus.cc:2258] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:11.073349 2676 ts_tablet_manager.cc:1905] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:11.074738 2677 tablet_replica.cc:330] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: stopping tablet replica
I20260620 00:12:11.074854 2677 raft_consensus.cc:2229] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:11.074911 2677 raft_consensus.cc:2258] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:11.075552 2676 ts_tablet_manager.cc:1918] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:11.075632 2676 log.cc:1196] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/a9eff746f7be4c1293fad3c51780da84
I20260620 00:12:11.075903 2676 ts_tablet_manager.cc:1939] T a9eff746f7be4c1293fad3c51780da84 P cc506c17ca474ec8a9e69402bda31131: Deleting consensus metadata
I20260620 00:12:11.076522 2106 catalog_manager.cc:4841] TS cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617): tablet a9eff746f7be4c1293fad3c51780da84 (table test-table [id=1fb9f459607d408c92e09e4929917f2c]) successfully deleted
I20260620 00:12:11.076695 2677 ts_tablet_manager.cc:1905] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:11.076936 2627 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: No bootstrap required, opened a new log
I20260620 00:12:11.076987 2627 ts_tablet_manager.cc:1397] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260620 00:12:11.077150 2630 tablet_bootstrap.cc:492] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: No bootstrap required, opened a new log
I20260620 00:12:11.077195 2630 ts_tablet_manager.cc:1397] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 00:12:11.077153 2627 raft_consensus.cc:348] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.077242 2627 raft_consensus.cc:374] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:11.077265 2627 raft_consensus.cc:729] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Initialized, Role: FOLLOWER
I20260620 00:12:11.077312 2627 consensus_queue.cc:260] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.077332 2630 raft_consensus.cc:348] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.077395 2630 raft_consensus.cc:374] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:11.077399 2627 ts_tablet_manager.cc:1428] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:12:11.077414 2630 raft_consensus.cc:729] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Initialized, Role: FOLLOWER
I20260620 00:12:11.077457 2630 consensus_queue.cc:260] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.077536 2630 ts_tablet_manager.cc:1428] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:11.079504 2677 ts_tablet_manager.cc:1918] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:11.079605 2677 log.cc:1196] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/a9eff746f7be4c1293fad3c51780da84
I20260620 00:12:11.079895 2677 ts_tablet_manager.cc:1939] T a9eff746f7be4c1293fad3c51780da84 P 3e8f735199d44431b1999081a6e8bff0: Deleting consensus metadata
I20260620 00:12:11.080367 2107 catalog_manager.cc:4841] TS 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875): tablet a9eff746f7be4c1293fad3c51780da84 (table test-table [id=1fb9f459607d408c92e09e4929917f2c]) successfully deleted
I20260620 00:12:11.169231 2674 raft_consensus.cc:482] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:11.169337 2674 raft_consensus.cc:504] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.169538 2674 leader_election.cc:288] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:11.169798 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1356ea92245ecaaa4b9e86d38a0a1" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0" is_pre_election: true
I20260620 00:12:11.169806 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1356ea92245ecaaa4b9e86d38a0a1" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349" is_pre_election: true
I20260620 00:12:11.169910 2418 raft_consensus.cc:2454] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:11.169909 2558 raft_consensus.cc:2454] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:11.170104 2210 leader_election.cc:302] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 896ecc145ee74d508be723558d569349, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:11.170238 2674 raft_consensus.cc:2781] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:11.170279 2674 raft_consensus.cc:482] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:11.170305 2674 raft_consensus.cc:3037] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:11.171074 2674 raft_consensus.cc:504] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.171239 2674 leader_election.cc:288] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:11.171442 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1356ea92245ecaaa4b9e86d38a0a1" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0"
I20260620 00:12:11.171491 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1356ea92245ecaaa4b9e86d38a0a1" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349"
I20260620 00:12:11.171550 2418 raft_consensus.cc:3037] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:11.171571 2558 raft_consensus.cc:3037] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:11.172170 2418 raft_consensus.cc:2454] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:11.172361 2211 leader_election.cc:302] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:11.172410 2558 raft_consensus.cc:2454] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:11.172597 2674 raft_consensus.cc:2781] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:11.172672 2674 raft_consensus.cc:686] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Becoming Leader. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Running, Role: LEADER
I20260620 00:12:11.172748 2674 consensus_queue.cc:237] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:11.173372 2120 catalog_manager.cc:5495] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc506c17ca474ec8a9e69402bda31131 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc506c17ca474ec8a9e69402bda31131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:11.191752 2418 raft_consensus.cc:1261] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.191735 2558 raft_consensus.cc:1261] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:11.191962 2674 consensus_queue.cc:1035] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:11.192124 2637 consensus_queue.cc:1035] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:12:11.196977 2657 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:12:12.715437 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:12.715592 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:12.716517 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Sending DeleteTablet for 3 replicas of tablet 03e1356ea92245ecaaa4b9e86d38a0a1
I20260620 00:12:12.716847 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41046
I20260620 00:12:12.716895 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46794
I20260620 00:12:12.717051 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1356ea92245ecaaa4b9e86d38a0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57856
I20260620 00:12:12.717092 2707 tablet_replica.cc:330] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: stopping tablet replica
I20260620 00:12:12.717149 2707 raft_consensus.cc:2229] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:12.717193 2707 raft_consensus.cc:2258] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.717242 2706 tablet_replica.cc:330] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: stopping tablet replica
I20260620 00:12:12.717305 2706 raft_consensus.cc:2229] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:12.717352 2706 raft_consensus.cc:2258] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.717478 2707 ts_tablet_manager.cc:1905] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.717592 2706 ts_tablet_manager.cc:1905] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.719206 2707 ts_tablet_manager.cc:1918] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:12:12.719287 2707 log.cc:1196] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/03e1356ea92245ecaaa4b9e86d38a0a1
I20260620 00:12:12.719522 2707 ts_tablet_manager.cc:1939] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 896ecc145ee74d508be723558d569349: Deleting consensus metadata
I20260620 00:12:12.719856 2708 tablet_replica.cc:330] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: stopping tablet replica
I20260620 00:12:12.719949 2708 raft_consensus.cc:2229] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:12.720118 2708 raft_consensus.cc:2258] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.720185 2106 catalog_manager.cc:4841] TS 896ecc145ee74d508be723558d569349 (127.0.98.3:46283): tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (table test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]) successfully deleted
I20260620 00:12:12.720373 2708 ts_tablet_manager.cc:1905] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.721498 2706 ts_tablet_manager.cc:1918] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:12:12.721577 2706 log.cc:1196] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/03e1356ea92245ecaaa4b9e86d38a0a1
I20260620 00:12:12.721815 2706 ts_tablet_manager.cc:1939] T 03e1356ea92245ecaaa4b9e86d38a0a1 P 3e8f735199d44431b1999081a6e8bff0: Deleting consensus metadata
I20260620 00:12:12.722229 2107 catalog_manager.cc:4841] TS 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875): tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (table test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]) successfully deleted
I20260620 00:12:12.722355 2708 ts_tablet_manager.cc:1918] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:12:12.722667 2708 log.cc:1196] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/03e1356ea92245ecaaa4b9e86d38a0a1
I20260620 00:12:12.722950 2708 ts_tablet_manager.cc:1939] T 03e1356ea92245ecaaa4b9e86d38a0a1 P cc506c17ca474ec8a9e69402bda31131: Deleting consensus metadata
I20260620 00:12:12.723296 2106 catalog_manager.cc:4841] TS cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617): tablet 03e1356ea92245ecaaa4b9e86d38a0a1 (table test-table [id=d4a9fe95595c400ca0bcccdd1e3bcf33]) successfully deleted
Jun 20 00:12:12 dist-test-slave-28bv krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:12:12 dist-test-slave-28bv krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914332, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:12.747769 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:12.735260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.0.98.62:44529)
0620 00:12:12.735396 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:12.735398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:12.735408 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:12.735474 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:12.735475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:12.735486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:12.735551 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:12.736124 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:12.736811 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:12.737518 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:12.737719 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:12.737769 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:12.745288 (+ 7519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:12.745290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:12.745290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:12.745299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:12.746836 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:12.747155 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:12.747155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:12.747156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:12.747209 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:12.747368 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:12.747368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:12.747369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:12.747476 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:12.747565 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:12.747634 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:12:12.747697 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260620 00:12:12.749392 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:12.749573 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:12.751093 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 (DEFAULT_TABLE table=test-table [id=62945ddac74445e1a27666580260fe41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.751102 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 (DEFAULT_TABLE table=test-table [id=62945ddac74445e1a27666580260fe41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.751120 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 (DEFAULT_TABLE table=test-table [id=62945ddac74445e1a27666580260fe41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.751247 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193f90d630fa492eaf8f8a832d9f9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.751247 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193f90d630fa492eaf8f8a832d9f9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.751257 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193f90d630fa492eaf8f8a832d9f9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.752753 2717 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Bootstrap starting.
I20260620 00:12:12.752925 2719 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Bootstrap starting.
I20260620 00:12:12.753176 2717 tablet_bootstrap.cc:654] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.753326 2719 tablet_bootstrap.cc:654] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.752758 2718 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Bootstrap starting.
I20260620 00:12:12.753818 2717 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: No bootstrap required, opened a new log
I20260620 00:12:12.753868 2717 ts_tablet_manager.cc:1397] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.753991 2717 raft_consensus.cc:348] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.754045 2718 tablet_bootstrap.cc:654] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.754092 2717 raft_consensus.cc:374] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.754118 2717 raft_consensus.cc:729] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.754142 2719 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: No bootstrap required, opened a new log
I20260620 00:12:12.754192 2719 ts_tablet_manager.cc:1397] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.754190 2717 consensus_queue.cc:260] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.754338 2719 raft_consensus.cc:348] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.754424 2719 raft_consensus.cc:374] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.754446 2719 raft_consensus.cc:729] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.754436 2717 ts_tablet_manager.cc:1428] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.754540 2719 consensus_queue.cc:260] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.754611 2718 tablet_bootstrap.cc:492] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: No bootstrap required, opened a new log
I20260620 00:12:12.754654 2718 ts_tablet_manager.cc:1397] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:12.754792 2718 raft_consensus.cc:348] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.754877 2718 raft_consensus.cc:374] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.754906 2718 raft_consensus.cc:729] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ecc145ee74d508be723558d569349, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.754895 2719 ts_tablet_manager.cc:1428] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.754948 2718 consensus_queue.cc:260] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.755168 2718 ts_tablet_manager.cc:1428] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:12.831089 2685 raft_consensus.cc:482] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:12.831321 2685 raft_consensus.cc:504] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.831568 2685 leader_election.cc:288] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:12.831792 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193f90d630fa492eaf8f8a832d9f9672" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349" is_pre_election: true
I20260620 00:12:12.831831 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193f90d630fa492eaf8f8a832d9f9672" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0" is_pre_election: true
I20260620 00:12:12.831919 2418 raft_consensus.cc:2454] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:12.831939 2558 raft_consensus.cc:2454] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:12.832096 2211 leader_election.cc:302] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:12.832211 2685 raft_consensus.cc:2781] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:12.832253 2685 raft_consensus.cc:482] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:12.832273 2685 raft_consensus.cc:3037] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:12.833129 2685 raft_consensus.cc:504] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.833334 2685 leader_election.cc:288] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:12.833508 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193f90d630fa492eaf8f8a832d9f9672" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0"
I20260620 00:12:12.833508 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193f90d630fa492eaf8f8a832d9f9672" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349"
I20260620 00:12:12.833603 2418 raft_consensus.cc:3037] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:12.833612 2558 raft_consensus.cc:3037] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:12.834453 2418 raft_consensus.cc:2454] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:12.834453 2558 raft_consensus.cc:2454] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:12.834646 2210 leader_election.cc:302] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 896ecc145ee74d508be723558d569349, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:12.834769 2685 raft_consensus.cc:2781] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:12.834836 2685 raft_consensus.cc:686] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Becoming Leader. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Running, Role: LEADER
I20260620 00:12:12.834892 2685 consensus_queue.cc:237] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.835647 2121 catalog_manager.cc:5495] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc506c17ca474ec8a9e69402bda31131 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc506c17ca474ec8a9e69402bda31131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:12.850811 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:12.848431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.0.98.1:37617)
0620 00:12:12.848623 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:12.848625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:12.848646 (+ 21us) server_negotiation.cc:407] Connection header received
0620 00:12:12.848668 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:12.848669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:12.848679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:12.848734 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:12.848872 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:12.849355 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:12.850145 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:12.850344 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:12.850412 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:12.850545 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:12.850574 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:12.850613 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:12.850667 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 00:12:12.850688 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":130,"threads_started":1}
I20260620 00:12:12.853480 2558 raft_consensus.cc:1261] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:12.853492 2418 raft_consensus.cc:1261] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:12.853696 2685 consensus_queue.cc:1035] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:12.853811 2698 consensus_queue.cc:1035] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:12.855350 2686 mvcc.cc:205] Tried to move back new op lower bound from 7298721107366600704 to 7298721107292409856. Current Snapshot: MvccSnapshot[applied={T|T < 7298721107366600704}]
I20260620 00:12:12.863677 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:12.863777 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:12.864647 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Sending DeleteTablet for 3 replicas of tablet 193f90d630fa492eaf8f8a832d9f9672
I20260620 00:12:12.864898 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41046
I20260620 00:12:12.865060 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46794
I20260620 00:12:12.865168 2706 tablet_replica.cc:330] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: stopping tablet replica
I20260620 00:12:12.865238 2706 raft_consensus.cc:2229] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:12.865290 2706 raft_consensus.cc:2258] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.865235 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 00:12:12.865371 2707 tablet_replica.cc:330] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: stopping tablet replica
W20260620 00:12:12.865391 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:12.865422 2707 raft_consensus.cc:2229] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:12.865468 2707 raft_consensus.cc:2258] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.865645 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 193f90d630fa492eaf8f8a832d9f9672 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57856
I20260620 00:12:12.865759 2708 tablet_replica.cc:330] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: stopping tablet replica
I20260620 00:12:12.865820 2708 raft_consensus.cc:2229] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:12.865821 2706 ts_tablet_manager.cc:1905] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.865878 2708 raft_consensus.cc:2258] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:12.865916 2707 ts_tablet_manager.cc:1905] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.866194 2708 ts_tablet_manager.cc:1905] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:12.867039 2538 tablet_service.cc:1467] Processing CreateTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 (DEFAULT_TABLE table=test-table [id=d9f8c444ee344b84a9872cc4970e6d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.867161 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96615932f1e4b8bbd5341754ed62ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.867826 2707 ts_tablet_manager.cc:1918] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:12.867826 2258 tablet_service.cc:1467] Processing CreateTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 (DEFAULT_TABLE table=test-table [id=d9f8c444ee344b84a9872cc4970e6d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.867851 2706 ts_tablet_manager.cc:1918] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:12.867877 2707 log.cc:1196] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/193f90d630fa492eaf8f8a832d9f9672
I20260620 00:12:12.867908 2706 log.cc:1196] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/193f90d630fa492eaf8f8a832d9f9672
I20260620 00:12:12.868009 2398 tablet_service.cc:1467] Processing CreateTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 (DEFAULT_TABLE table=test-table [id=d9f8c444ee344b84a9872cc4970e6d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:12.868112 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96615932f1e4b8bbd5341754ed62ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.868134 2706 ts_tablet_manager.cc:1939] T 193f90d630fa492eaf8f8a832d9f9672 P 3e8f735199d44431b1999081a6e8bff0: Deleting consensus metadata
I20260620 00:12:12.868144 2707 ts_tablet_manager.cc:1939] T 193f90d630fa492eaf8f8a832d9f9672 P 896ecc145ee74d508be723558d569349: Deleting consensus metadata
I20260620 00:12:12.868476 2106 catalog_manager.cc:4841] TS 896ecc145ee74d508be723558d569349 (127.0.98.3:46283): tablet 193f90d630fa492eaf8f8a832d9f9672 (table test-table [id=62945ddac74445e1a27666580260fe41]) successfully deleted
I20260620 00:12:12.869280 2717 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: Bootstrap starting.
I20260620 00:12:12.869352 2718 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Bootstrap starting.
I20260620 00:12:12.869405 2107 catalog_manager.cc:4841] TS 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875): tablet 193f90d630fa492eaf8f8a832d9f9672 (table test-table [id=62945ddac74445e1a27666580260fe41]) successfully deleted
I20260620 00:12:12.869602 2708 ts_tablet_manager.cc:1918] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:12:12.869638 2717 tablet_bootstrap.cc:654] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.869637 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96615932f1e4b8bbd5341754ed62ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:12.869653 2708 log.cc:1196] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/193f90d630fa492eaf8f8a832d9f9672
I20260620 00:12:12.869818 2718 tablet_bootstrap.cc:654] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.869918 2708 ts_tablet_manager.cc:1939] T 193f90d630fa492eaf8f8a832d9f9672 P cc506c17ca474ec8a9e69402bda31131: Deleting consensus metadata
I20260620 00:12:12.870401 2106 catalog_manager.cc:4841] TS cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617): tablet 193f90d630fa492eaf8f8a832d9f9672 (table test-table [id=62945ddac74445e1a27666580260fe41]) successfully deleted
I20260620 00:12:12.870483 2717 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: No bootstrap required, opened a new log
I20260620 00:12:12.870527 2717 ts_tablet_manager.cc:1397] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.870690 2717 raft_consensus.cc:348] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.870770 2717 raft_consensus.cc:374] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.870795 2717 raft_consensus.cc:729] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e8f735199d44431b1999081a6e8bff0, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.870857 2717 consensus_queue.cc:260] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.870966 2717 ts_tablet_manager.cc:1428] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:12.871030 2719 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131: Bootstrap starting.
I20260620 00:12:12.871353 2719 tablet_bootstrap.cc:654] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:12.871495 2718 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: No bootstrap required, opened a new log
I20260620 00:12:12.871549 2718 ts_tablet_manager.cc:1397] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:12.871703 2718 raft_consensus.cc:348] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.871758 2718 raft_consensus.cc:374] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.871803 2718 raft_consensus.cc:729] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896ecc145ee74d508be723558d569349, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.871853 2718 consensus_queue.cc:260] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.871958 2719 tablet_bootstrap.cc:492] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131: No bootstrap required, opened a new log
I20260620 00:12:12.871958 2718 ts_tablet_manager.cc:1428] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:12.871995 2719 ts_tablet_manager.cc:1397] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:12.872113 2719 raft_consensus.cc:348] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.872176 2719 raft_consensus.cc:374] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:12.872190 2719 raft_consensus.cc:729] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Initialized, Role: FOLLOWER
I20260620 00:12:12.872252 2719 consensus_queue.cc:260] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:12.872340 2719 ts_tablet_manager.cc:1428] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:13.005241 2730 raft_consensus.cc:482] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:13.005367 2730 raft_consensus.cc:504] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:13.005563 2730 leader_election.cc:288] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:13.005841 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96615932f1e4b8bbd5341754ed62ae4" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349" is_pre_election: true
I20260620 00:12:13.005874 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96615932f1e4b8bbd5341754ed62ae4" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0" is_pre_election: true
I20260620 00:12:13.005959 2558 raft_consensus.cc:2454] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:13.005978 2418 raft_consensus.cc:2454] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 0.
I20260620 00:12:13.006147 2210 leader_election.cc:302] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 896ecc145ee74d508be723558d569349, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:13.006268 2730 raft_consensus.cc:2781] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:13.006347 2730 raft_consensus.cc:482] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:13.006389 2730 raft_consensus.cc:3037] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:13.007210 2730 raft_consensus.cc:504] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:13.007378 2730 leader_election.cc:288] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Requested vote from peers 3e8f735199d44431b1999081a6e8bff0 (127.0.98.2:39875), 896ecc145ee74d508be723558d569349 (127.0.98.3:46283)
I20260620 00:12:13.007597 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96615932f1e4b8bbd5341754ed62ae4" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e8f735199d44431b1999081a6e8bff0"
I20260620 00:12:13.007686 2418 raft_consensus.cc:3037] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:13.007668 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a96615932f1e4b8bbd5341754ed62ae4" candidate_uuid: "cc506c17ca474ec8a9e69402bda31131" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "896ecc145ee74d508be723558d569349"
I20260620 00:12:13.007787 2558 raft_consensus.cc:3037] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:13.008359 2418 raft_consensus.cc:2454] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:13.008513 2558 raft_consensus.cc:2454] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc506c17ca474ec8a9e69402bda31131 in term 1.
I20260620 00:12:13.008610 2211 leader_election.cc:302] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e8f735199d44431b1999081a6e8bff0, cc506c17ca474ec8a9e69402bda31131; no voters:
I20260620 00:12:13.008708 2730 raft_consensus.cc:2781] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:13.008883 2730 raft_consensus.cc:686] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [term 1 LEADER]: Becoming Leader. State: Replica: cc506c17ca474ec8a9e69402bda31131, State: Running, Role: LEADER
I20260620 00:12:13.008955 2730 consensus_queue.cc:237] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } }
I20260620 00:12:13.009626 2119 catalog_manager.cc:5495] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc506c17ca474ec8a9e69402bda31131 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc506c17ca474ec8a9e69402bda31131" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc506c17ca474ec8a9e69402bda31131" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:13.023679 2418 raft_consensus.cc:1261] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:13.023679 2558 raft_consensus.cc:1261] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Refusing update from remote peer cc506c17ca474ec8a9e69402bda31131: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:13.023921 2730 consensus_queue.cc:1035] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "896ecc145ee74d508be723558d569349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:13.024050 2685 consensus_queue.cc:1035] T a96615932f1e4b8bbd5341754ed62ae4 P cc506c17ca474ec8a9e69402bda31131 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e8f735199d44431b1999081a6e8bff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:12:13.028707 2691 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:12:14.546599 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:14.546701 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:14.547523 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e3ff66409834a3793d6f3fdb16b41df: Sending DeleteTablet for 3 replicas of tablet a96615932f1e4b8bbd5341754ed62ae4
I20260620 00:12:14.547829 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46794
I20260620 00:12:14.547852 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57856
I20260620 00:12:14.547991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2193
I20260620 00:12:14.548019 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet a96615932f1e4b8bbd5341754ed62ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41046
I20260620 00:12:14.549757 2759 tablet_replica.cc:330] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: stopping tablet replica
I20260620 00:12:14.549808 2759 raft_consensus.cc:2229] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:14.549834 2759 raft_consensus.cc:2258] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:14.550093 2759 ts_tablet_manager.cc:1905] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:14.551826 2759 ts_tablet_manager.cc:1918] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:12:14.551879 2759 log.cc:1196] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/a96615932f1e4b8bbd5341754ed62ae4
I20260620 00:12:14.552129 2759 ts_tablet_manager.cc:1939] T a96615932f1e4b8bbd5341754ed62ae4 P 896ecc145ee74d508be723558d569349: Deleting consensus metadata
I20260620 00:12:14.552477 2106 catalog_manager.cc:4841] TS 896ecc145ee74d508be723558d569349 (127.0.98.3:46283): tablet a96615932f1e4b8bbd5341754ed62ae4 (table test-table [id=d9f8c444ee344b84a9872cc4970e6d0c]) successfully deleted
I20260620 00:12:14.552685 2761 tablet_replica.cc:330] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: stopping tablet replica
I20260620 00:12:14.552742 2761 raft_consensus.cc:2229] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:14.552802 2761 raft_consensus.cc:2258] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:14.553005 2761 ts_tablet_manager.cc:1905] T a96615932f1e4b8bbd5341754ed62ae4 P 3e8f735199d44431b1999081a6e8bff0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:12:14.554421 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37617 (error 108)
I20260620 00:12:14.554641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2333
I20260620 00:12:14.555495 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:14.555294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37617 (local address 127.0.0.1:57868)
0620 00:12:14.555418 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:14.555462 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
W20260620 00:12:14.555612 2106 catalog_manager.cc:4584] TS cc506c17ca474ec8a9e69402bda31131 (127.0.98.1:37617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a96615932f1e4b8bbd5341754ed62ae4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37617: connect: Connection refused (error 111)
I20260620 00:12:14.561444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2473
I20260620 00:12:14.561518 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:14.561399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39875 (local address 127.0.0.1:41072)
0620 00:12:14.561448 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:14.561481 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 00:12:14.566382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2089
2026-06-20T00:12:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5945 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:14 dist-test-slave-28bv krb5kdc[2766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:14 dist-test-slave-28bv krb5kdc[2766](info): set up 2 sockets
Jun 20 00:12:14 dist-test-slave-28bv krb5kdc[2766](info): commencing operation
krb5kdc: starting...
W20260620 00:12:15.624333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:15 dist-test-slave-28bv krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:15.747887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:15.829526 2779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:15.829806 2779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:15.829845 2779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:15.831091 2779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:15.831173 2779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:15.831198 2779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:15.831219 2779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:15.831277 2779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:15.832954 2779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:15.833258 2779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:15.833506 2779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:15.839248 2779 server_base.cc:1034] running on GCE node
W20260620 00:12:15.839551 2785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:15.839756 2784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:15.839938 2787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:15.840226 2779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:15.840601 2779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:15.841805 2779 hybrid_clock.cc:648] HybridClock initialized: now 1781914335841776 us; error 42 us; skew 500 ppm
Jun 20 00:12:15 dist-test-slave-28bv krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:15.844624 2779 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:15.845266 2779 webserver.cc:460] Webserver started at http://127.0.98.62:39213/ using document root <none> and password file <none>
I20260620 00:12:15.845528 2779 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:15.845602 2779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:15.845765 2779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:15.846865 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "833e7c8e5c844ffe84741180e45ea488"
format_stamp: "Formatted at 2026-06-20 00:12:15 on dist-test-slave-28bv"
server_key: "16ae6762e9d62ff471a33c1c2b2c15bf"
server_key_iv: "ba28e3c16026e925e8d3d6ae4740f819"
server_key_version: "encryptionkey@0"
I20260620 00:12:15.847237 2779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "833e7c8e5c844ffe84741180e45ea488"
format_stamp: "Formatted at 2026-06-20 00:12:15 on dist-test-slave-28bv"
server_key: "16ae6762e9d62ff471a33c1c2b2c15bf"
server_key_iv: "ba28e3c16026e925e8d3d6ae4740f819"
server_key_version: "encryptionkey@0"
I20260620 00:12:15.848722 2779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:12:15.849520 2794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:15.849723 2779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:15.849835 2779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "833e7c8e5c844ffe84741180e45ea488"
format_stamp: "Formatted at 2026-06-20 00:12:15 on dist-test-slave-28bv"
server_key: "16ae6762e9d62ff471a33c1c2b2c15bf"
server_key_iv: "ba28e3c16026e925e8d3d6ae4740f819"
server_key_version: "encryptionkey@0"
I20260620 00:12:15.849913 2779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:15.862630 2779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:15.865157 2779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:15.865346 2779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:15.869727 2779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38657
I20260620 00:12:15.869820 2846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38657 every 8 connection(s)
I20260620 00:12:15.870338 2779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:15.871024 2847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:15.873076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2779
I20260620 00:12:15.873169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:15.873338 392 external_mini_cluster.cc:1428] Setting key 3c844d48c3fc05de5b89163601063f95
I20260620 00:12:15.873863 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: Bootstrap starting.
I20260620 00:12:15.874513 2847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:15.874892 2847 log.cc:824] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:15.875689 2847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: No bootstrap required, opened a new log
I20260620 00:12:15.878000 2847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260620 00:12:15.878186 2847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:15.878234 2847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 833e7c8e5c844ffe84741180e45ea488, State: Initialized, Role: FOLLOWER
I20260620 00:12:15.878422 2847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260620 00:12:15.878520 2847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:15.878566 2847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:15.878615 2847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:15.879220 2847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260620 00:12:15.879354 2847 leader_election.cc:302] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 833e7c8e5c844ffe84741180e45ea488; no voters:
I20260620 00:12:15.879576 2847 leader_election.cc:288] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:15.879604 2852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:15.879720 2852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [term 1 LEADER]: Becoming Leader. State: Replica: 833e7c8e5c844ffe84741180e45ea488, State: Running, Role: LEADER
I20260620 00:12:15.879850 2852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260620 00:12:15.879899 2847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:15.880187 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "833e7c8e5c844ffe84741180e45ea488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } } }
I20260620 00:12:15.880355 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:15.880574 2852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 833e7c8e5c844ffe84741180e45ea488. Latest consensus state: current_term: 1 leader_uuid: "833e7c8e5c844ffe84741180e45ea488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "833e7c8e5c844ffe84741180e45ea488" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } } }
I20260620 00:12:15.880712 2852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488 [sys.catalog]: This master's current role is: LEADER
Jun 20 00:12:15 dist-test-slave-28bv krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260620 00:12:15.881979 2868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:15.882054 2868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:15.882112 2859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:15.882356 2859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:15.883940 2859 catalog_manager.cc:1347] Generated new cluster ID: bdcaaf13c2974013853aa8148b5f77c3
I20260620 00:12:15.883996 2859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:15.886308 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:15.873890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.0.98.62:38657)
0620 00:12:15.874066 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:15.874075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:15.874102 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:12:15.874530 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:15.874537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:15.874585 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:15.874760 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:15.875604 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:15.876775 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:15.877951 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:15.878507 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:15.878643 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:15.881727 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:15.881732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:15.881732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:15.881746 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:15.883902 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:15.884347 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:15.884350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:15.884351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:15.884461 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:15.884714 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:15.884715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:15.884716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:15.885044 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:15.885200 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:15.885773 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 00:12:15.886071 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260620 00:12:15.890311 2859 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:15.891007 2859 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:15.897318 2859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 833e7c8e5c844ffe84741180e45ea488: Generated new TSK 0
I20260620 00:12:15.897528 2859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:15.932689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:16.026546 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:16.026788 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:16.026815 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:16.028088 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:16.028138 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:16.028168 2873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:16.029912 2873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:16.030195 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:16.030448 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:16.036732 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:16.036764 2873 server_base.cc:1034] running on GCE node
W20260620 00:12:16.037025 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:16.036736 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:16.037349 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:16.037669 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:16.038904 2873 hybrid_clock.cc:648] HybridClock initialized: now 1781914336038866 us; error 49 us; skew 500 ppm
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:16.041848 2873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:16.042393 2873 webserver.cc:460] Webserver started at http://127.0.98.1:44967/ using document root <none> and password file <none>
I20260620 00:12:16.042615 2873 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:16.042658 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:16.042755 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:16.044230 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "2cac1dc85dff47fdb67c98b2a2e6f414"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "062e828404b094d85590e41ec6b8ab9b"
server_key_iv: "3ca1b75a49f5273aa73406fa0e97a188"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.044694 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2cac1dc85dff47fdb67c98b2a2e6f414"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "062e828404b094d85590e41ec6b8ab9b"
server_key_iv: "3ca1b75a49f5273aa73406fa0e97a188"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.046211 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:16.047363 2888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.047991 2873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:16.048527 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "2cac1dc85dff47fdb67c98b2a2e6f414"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "062e828404b094d85590e41ec6b8ab9b"
server_key_iv: "3ca1b75a49f5273aa73406fa0e97a188"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.048733 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:16.086118 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:16.088762 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:16.088930 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:16.089196 2873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:16.089545 2873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:16.089600 2873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.089644 2873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:16.089671 2873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.095808 2873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39585
I20260620 00:12:16.095891 3001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39585 every 8 connection(s)
I20260620 00:12:16.096243 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:16.099031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 2873
I20260620 00:12:16.099171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:16.099359 392 external_mini_cluster.cc:1428] Setting key 2c04a8ae2e9abef27fbace34ec9281b1
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:12:16.108520 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.097183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57327 (local address 127.0.98.62:38657)
0620 00:12:16.097269 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:16.097272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:16.097459 (+ 187us) server_negotiation.cc:407] Connection header received
0620 00:12:16.098522 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:16.098523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:16.098537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:16.098606 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:16.099451 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.100023 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.101286 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.101521 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.101620 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.104799 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:16.104802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:16.104802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:16.104813 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:16.106803 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.107350 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.107351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.107352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.107423 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.107731 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.107732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.107732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.107866 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:16.107956 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:16.108307 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 00:12:16.108418 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 00:12:16.108767 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.097100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38657 (local address 127.0.98.1:57327)
0620 00:12:16.097306 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:16.097350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:16.098394 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:16.098698 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:16.098702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:16.099022 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:16.099344 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.099350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.100145 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.100146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.101183 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.101186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.101632 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.101634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.101777 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.102445 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:16.102458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:16.104656 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:16.106946 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.106959 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.106962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.107246 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.107514 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.107516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.107516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.107657 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.107946 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:16.107954 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:16.108153 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 00:12:16.108273 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:12:16.108473 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260620 00:12:16.109504 3002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38657
I20260620 00:12:16.109644 3002 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:16.109880 3002 heartbeater.cc:511] Master 127.0.98.62:38657 requested a full tablet report, sending...
I20260620 00:12:16.110546 2811 ts_manager.cc:194] Registered new tserver with Master: 2cac1dc85dff47fdb67c98b2a2e6f414 (127.0.98.1:39585)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:12:16.111400 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:16.128196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:16.215709 3007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:16.215942 3007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:16.215969 3007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:16.217312 3007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:16.217365 3007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:16.217396 3007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:16.219038 3007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:16.219324 3007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:16.219573 3007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:16.224897 3015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:16.225028 3007 server_base.cc:1034] running on GCE node
W20260620 00:12:16.224897 3013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:16.225015 3012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:16.225512 3007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:16.225839 3007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:16.227044 3007 hybrid_clock.cc:648] HybridClock initialized: now 1781914336227032 us; error 55 us; skew 500 ppm
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:16.229880 3007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:16.230434 3007 webserver.cc:460] Webserver started at http://127.0.98.2:37337/ using document root <none> and password file <none>
I20260620 00:12:16.230693 3007 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:16.230765 3007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:16.230919 3007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:16.232045 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "45606b66571244228d50f2b85b69706f"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "52a27aedcdb2dfff263e7bf8a6214743"
server_key_iv: "b4cea3ba062c88e01f2ce945883d1fe8"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.232424 3007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45606b66571244228d50f2b85b69706f"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "52a27aedcdb2dfff263e7bf8a6214743"
server_key_iv: "b4cea3ba062c88e01f2ce945883d1fe8"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.233912 3007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:16.234783 3022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.235052 3007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:16.235188 3007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "45606b66571244228d50f2b85b69706f"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "52a27aedcdb2dfff263e7bf8a6214743"
server_key_iv: "b4cea3ba062c88e01f2ce945883d1fe8"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.235297 3007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:16.268196 3007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:16.270614 3007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:16.270805 3007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:16.271056 3007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:16.271389 3007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:16.271443 3007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.271485 3007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:16.271514 3007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.277671 3007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40691
I20260620 00:12:16.277717 3135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40691 every 8 connection(s)
I20260620 00:12:16.278072 3007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:16.283995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3007
I20260620 00:12:16.284134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:16.284353 392 external_mini_cluster.cc:1428] Setting key 788850c7e798f5d50c1451d28c0b6d69
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:16.289552 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.278923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48627 (local address 127.0.98.62:38657)
0620 00:12:16.279035 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:16.279037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:16.279209 (+ 172us) server_negotiation.cc:407] Connection header received
0620 00:12:16.280339 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:16.280341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:16.280354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:16.280409 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:16.281166 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.281675 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.282777 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.282979 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.283095 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.285878 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:16.285880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:16.285880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:16.285888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:16.287420 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.287973 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.287975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.287975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.288073 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.288522 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.288523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.288523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.288794 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:16.288876 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:16.289394 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 00:12:16.289481 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 00:12:16.289978 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.278836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38657 (local address 127.0.98.2:48627)
0620 00:12:16.279067 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:16.279118 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:16.280198 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:16.280495 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:16.280499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:16.280858 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:16.281059 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.281064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.281794 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.281795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.282669 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.282671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.283108 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.283109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.283254 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.283864 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:16.283873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:16.285753 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:16.287514 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.287522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.287525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.287872 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.288159 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.288160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.288160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.288440 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.288937 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:16.288946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:16.289202 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 00:12:16.289361 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 00:12:16.289584 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260620 00:12:16.290611 3136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38657
I20260620 00:12:16.290730 3136 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:16.290954 3136 heartbeater.cc:511] Master 127.0.98.62:38657 requested a full tablet report, sending...
I20260620 00:12:16.291363 2811 ts_manager.cc:194] Registered new tserver with Master: 45606b66571244228d50f2b85b69706f (127.0.98.2:40691)
I20260620 00:12:16.291740 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48627
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:16.315680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:16.400877 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:16.401142 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:16.401185 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:16.402527 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:16.402614 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:16.402663 3141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:16.404414 3141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:16.404754 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:16.405020 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:16.410645 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:16.410646 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:16.410698 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:16.411278 3141 server_base.cc:1034] running on GCE node
I20260620 00:12:16.411571 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:16.411895 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:16.413038 3141 hybrid_clock.cc:648] HybridClock initialized: now 1781914336413024 us; error 33 us; skew 500 ppm
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:16.415951 3141 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:16.416468 3141 webserver.cc:460] Webserver started at http://127.0.98.3:34425/ using document root <none> and password file <none>
I20260620 00:12:16.416742 3141 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:16.416795 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:16.416968 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:16.418118 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "6075f506686d474cb790e0435a6d1f10"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "719e569d635c9699d4a679810b6827bd"
server_key_iv: "323b8e5a24f9d82bc7838078191cd60f"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.418499 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6075f506686d474cb790e0435a6d1f10"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "719e569d635c9699d4a679810b6827bd"
server_key_iv: "323b8e5a24f9d82bc7838078191cd60f"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.419984 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:16.421056 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.421367 3141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:16.421487 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "6075f506686d474cb790e0435a6d1f10"
format_stamp: "Formatted at 2026-06-20 00:12:16 on dist-test-slave-28bv"
server_key: "719e569d635c9699d4a679810b6827bd"
server_key_iv: "323b8e5a24f9d82bc7838078191cd60f"
server_key_version: "encryptionkey@0"
I20260620 00:12:16.421578 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:16.434556 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:16.437463 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:16.437650 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:16.437937 3141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:16.438289 3141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:16.438333 3141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.438356 3141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:16.438369 3141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:16.445465 3141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37977
I20260620 00:12:16.445530 3269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37977 every 8 connection(s)
I20260620 00:12:16.445914 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:16.451257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3141
I20260620 00:12:16.451383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:16.451597 392 external_mini_cluster.cc:1428] Setting key 5bb47cb74976bcb3fe8c53ab21420d97
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[2766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:16.458415 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.446902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46489 (local address 127.0.98.62:38657)
0620 00:12:16.446976 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:16.446978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:16.447149 (+ 171us) server_negotiation.cc:407] Connection header received
0620 00:12:16.448216 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:16.448217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:16.448230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:16.448304 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:16.449428 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.449950 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.451530 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.451865 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.451947 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.455035 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:16.455037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:16.455037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:16.455046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:16.456586 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.457159 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.457161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.457162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.457233 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:16.457610 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:16.457610 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:16.457611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:16.457753 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:16.457836 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:16.458251 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 00:12:16.458344 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:12:16.458560 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:16.446849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38657 (local address 127.0.98.3:46489)
0620 00:12:16.447011 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:16.447055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:16.448008 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:16.448436 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:16.448441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:16.449053 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:16.449281 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.449287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.450080 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.450082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.451361 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:16.451365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:16.451997 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:16.451998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:16.452234 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:16.453053 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:16.453062 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:16.454896 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:16.456769 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.456777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.456780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.457060 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.457312 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:16.457313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:16.457313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:16.457437 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:16.457874 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:16.457880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:16.458029 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 00:12:16.458158 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 00:12:16.458327 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260620 00:12:16.459230 3270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38657
I20260620 00:12:16.459352 3270 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:16.459570 3270 heartbeater.cc:511] Master 127.0.98.62:38657 requested a full tablet report, sending...
I20260620 00:12:16.460180 2811 ts_manager.cc:194] Registered new tserver with Master: 6075f506686d474cb790e0435a6d1f10 (127.0.98.3:37977)
I20260620 00:12:16.460654 2811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46489
I20260620 00:12:16.464519 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:12:16.476467 2850 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:16.475237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.98.62:38657)
0620 00:12:16.475329 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:16.475332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:16.475430 (+ 98us) server_negotiation.cc:407] Connection header received
0620 00:12:16.475537 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:16.475538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:16.475549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:16.475697 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:16.476382 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55552: BlockingRecv error: recv got EOF from 127.0.0.1:55552 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:12:16.477386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2873
I20260620 00:12:16.482604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3007
I20260620 00:12:16.487609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3141
I20260620 00:12:16.492254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 2779
2026-06-20T00:12:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1924 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[3282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[3282](info): set up 2 sockets
Jun 20 00:12:16 dist-test-slave-28bv krb5kdc[3282](info): commencing operation
krb5kdc: starting...
W20260620 00:12:17.532388 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:17.655254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:17.736588 3295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:17.736874 3295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:17.736913 3295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:17.738205 3295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:17.738291 3295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:17.738314 3295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:17.738336 3295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:17.738369 3295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:17.740080 3295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:17.740402 3295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:17.740716 3295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:17.746140 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:17.746151 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:17.746259 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:17.746475 3295 server_base.cc:1034] running on GCE node
I20260620 00:12:17.746768 3295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:17.747049 3295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:17.748221 3295 hybrid_clock.cc:648] HybridClock initialized: now 1781914337748197 us; error 37 us; skew 500 ppm
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:17.755177 3295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:17.755909 3295 webserver.cc:460] Webserver started at http://127.0.98.62:36873/ using document root <none> and password file <none>
I20260620 00:12:17.756196 3295 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:17.756276 3295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:17.756433 3295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:17.757670 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "529b5b05a8f24477a992ca680a86b0a7"
format_stamp: "Formatted at 2026-06-20 00:12:17 on dist-test-slave-28bv"
server_key: "dd7031e13c70fb4866237185738f2daa"
server_key_iv: "b340b31790b284c70b01337e6ff6cbcc"
server_key_version: "encryptionkey@0"
I20260620 00:12:17.758062 3295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "529b5b05a8f24477a992ca680a86b0a7"
format_stamp: "Formatted at 2026-06-20 00:12:17 on dist-test-slave-28bv"
server_key: "dd7031e13c70fb4866237185738f2daa"
server_key_iv: "b340b31790b284c70b01337e6ff6cbcc"
server_key_version: "encryptionkey@0"
I20260620 00:12:17.759509 3295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:17.760509 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:17.760701 3295 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:17.760823 3295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "529b5b05a8f24477a992ca680a86b0a7"
format_stamp: "Formatted at 2026-06-20 00:12:17 on dist-test-slave-28bv"
server_key: "dd7031e13c70fb4866237185738f2daa"
server_key_iv: "b340b31790b284c70b01337e6ff6cbcc"
server_key_version: "encryptionkey@0"
I20260620 00:12:17.760919 3295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:17.782241 3295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:17.784494 3295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:17.784715 3295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:17.789247 3295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44705
I20260620 00:12:17.789304 3362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44705 every 8 connection(s)
I20260620 00:12:17.789707 3295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:17.790426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3295
I20260620 00:12:17.790534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:17.790717 392 external_mini_cluster.cc:1428] Setting key f75a1bcb165ad1624c095baf59a50780
I20260620 00:12:17.790701 3363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:17.794409 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7: Bootstrap starting.
I20260620 00:12:17.795179 3363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:17.795477 3363 log.cc:824] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:17.796373 3363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7: No bootstrap required, opened a new log
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:17.798913 3363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } }
I20260620 00:12:17.799116 3363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:17.799155 3363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529b5b05a8f24477a992ca680a86b0a7, State: Initialized, Role: FOLLOWER
I20260620 00:12:17.799300 3363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } }
I20260620 00:12:17.799381 3363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:17.799425 3363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:17.799500 3363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:17.800271 3363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } }
I20260620 00:12:17.800387 3363 leader_election.cc:302] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 529b5b05a8f24477a992ca680a86b0a7; no voters:
I20260620 00:12:17.800619 3363 leader_election.cc:288] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:17.800750 3368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:17.800902 3363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:17.800927 3368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 529b5b05a8f24477a992ca680a86b0a7, State: Running, Role: LEADER
I20260620 00:12:17.801026 3368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } }
I20260620 00:12:17.801357 3368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529b5b05a8f24477a992ca680a86b0a7. Latest consensus state: current_term: 1 leader_uuid: "529b5b05a8f24477a992ca680a86b0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } } }
I20260620 00:12:17.801455 3368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:17.801369 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "529b5b05a8f24477a992ca680a86b0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529b5b05a8f24477a992ca680a86b0a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44705 } } }
I20260620 00:12:17.801565 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:17.802080 3375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:17.802580 3375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:17.802742 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:17.791284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.62:44705)
0620 00:12:17.791461 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:17.791469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:17.791504 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:12:17.792038 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:17.792046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:17.792097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:17.792243 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:17.793104 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:17.794018 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:17.795061 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:17.795550 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:17.795674 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:17.798343 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:17.798349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:17.798349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:17.798367 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:17.800604 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:17.801072 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:17.801075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:17.801076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:17.801187 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:17.801421 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:17.801422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:17.801423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:17.801755 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:17.802060 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:17.802307 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 00:12:17.802451 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260620 00:12:17.804080 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:12:17.804394 3375 catalog_manager.cc:1347] Generated new cluster ID: 2057986ae25c48588ed4f56a99878122
I20260620 00:12:17.804445 3375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:17.821480 3366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:17.817706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.62:44705)
0620 00:12:17.817756 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:17.817759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:17.817935 (+ 176us) server_negotiation.cc:407] Connection header received
0620 00:12:17.818012 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:17.818013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:17.818033 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:17.818104 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:17.818494 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:17.818979 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:17.819743 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:17.819964 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:17.820051 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:17.820832 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:17.820834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:17.820834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:17.821109 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:17.821130 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:17.821161 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:17.821213 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:17.821300 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 00:12:17.821323 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:12:17.827190 3375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:17.827937 3375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:17.841902 3375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 529b5b05a8f24477a992ca680a86b0a7: Generated new TSK 0
I20260620 00:12:17.842159 3375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:12:17.844982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3295
2026-06-20T00:12:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1350 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3396](info): set up 2 sockets
Jun 20 00:12:17 dist-test-slave-28bv krb5kdc[3396](info): commencing operation
krb5kdc: starting...
W20260620 00:12:18.892814 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:18 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:19.020776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:19.103871 3409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:19.104120 3409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:19.104146 3409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:19.105392 3409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:19.105453 3409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:19.105475 3409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:19.105504 3409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:19.105516 3409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:19.107239 3409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:19.107488 3409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:19.107676 3409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:19.112699 3417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.112811 3409 server_base.cc:1034] running on GCE node
W20260620 00:12:19.114888 3415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.115311 3414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.115751 3409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:19.116096 3409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:19.117268 3409 hybrid_clock.cc:648] HybridClock initialized: now 1781914339117237 us; error 51 us; skew 500 ppm
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:19.120115 3409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:19.120747 3409 webserver.cc:460] Webserver started at http://127.0.98.62:39287/ using document root <none> and password file <none>
I20260620 00:12:19.120990 3409 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:19.121066 3409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:19.121217 3409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:19.122334 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "4f488b1edcc040689fd3c5196167ec02"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "b7a1365d6381da119962a1c64009d080"
server_key_iv: "61f6e6d9bf00bd5bfff1e8eb7b49893a"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.122715 3409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f488b1edcc040689fd3c5196167ec02"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "b7a1365d6381da119962a1c64009d080"
server_key_iv: "61f6e6d9bf00bd5bfff1e8eb7b49893a"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.124171 3409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:19.125135 3424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.125385 3409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:19.125557 3409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "4f488b1edcc040689fd3c5196167ec02"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "b7a1365d6381da119962a1c64009d080"
server_key_iv: "61f6e6d9bf00bd5bfff1e8eb7b49893a"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.125644 3409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:19.151742 3409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:19.154335 3409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:19.154527 3409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:19.158866 3409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44707
I20260620 00:12:19.158910 3476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44707 every 8 connection(s)
I20260620 00:12:19.159276 3409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:19.160328 3477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:19.162779 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02: Bootstrap starting.
I20260620 00:12:19.163425 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:19.163724 3477 log.cc:824] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:19.164433 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02: No bootstrap required, opened a new log
I20260620 00:12:19.166075 3477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } }
I20260620 00:12:19.166232 3477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:19.166255 3477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f488b1edcc040689fd3c5196167ec02, State: Initialized, Role: FOLLOWER
I20260620 00:12:19.166366 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } }
I20260620 00:12:19.166467 3477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:19.166507 3477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:19.166599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3409
I20260620 00:12:19.166599 3477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:19.166692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:19.166880 392 external_mini_cluster.cc:1428] Setting key 9d8b1c7749abf03bb3488bec6a23faaa
I20260620 00:12:19.167806 3477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } }
I20260620 00:12:19.167904 3477 leader_election.cc:302] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f488b1edcc040689fd3c5196167ec02; no voters:
I20260620 00:12:19.168124 3477 leader_election.cc:288] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:19.168146 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:19.168265 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [term 1 LEADER]: Becoming Leader. State: Replica: 4f488b1edcc040689fd3c5196167ec02, State: Running, Role: LEADER
I20260620 00:12:19.168355 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } }
I20260620 00:12:19.168409 3477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:19.168706 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f488b1edcc040689fd3c5196167ec02. Latest consensus state: current_term: 1 leader_uuid: "4f488b1edcc040689fd3c5196167ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } } }
I20260620 00:12:19.168813 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:19.168992 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f488b1edcc040689fd3c5196167ec02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f488b1edcc040689fd3c5196167ec02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44707 } } }
I20260620 00:12:19.169059 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:19.169390 3487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:19.169975 3487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:19.171871 3487 catalog_manager.cc:1347] Generated new cluster ID: 3ff8029115fe467ea02214fe94480d17
I20260620 00:12:19.171931 3487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:19.180681 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.167505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.0.98.62:44707)
0620 00:12:19.167695 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:19.167705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:19.167743 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:12:19.167847 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:19.167854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:19.167935 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:19.168103 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:19.169392 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.170830 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.172691 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.173373 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.173502 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.176573 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:19.176579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:19.176579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:19.176594 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:19.178760 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.179252 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.179256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.179256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.179388 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.179636 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.179636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.179637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.179970 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:19.180133 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:19.180251 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 00:12:19.180396 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260620 00:12:19.183211 3487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:19.183791 3487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:19.191582 3487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f488b1edcc040689fd3c5196167ec02: Generated new TSK 0
I20260620 00:12:19.191804 3487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:19.221045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:19.314055 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:19.314296 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:19.314324 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:19.315740 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:19.315804 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:19.315836 3503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:19.317721 3503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:19.318053 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:19.318333 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:19.324146 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.324151 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.324312 3503 server_base.cc:1034] running on GCE node
W20260620 00:12:19.324151 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.324945 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:19.325260 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:19.326508 3503 hybrid_clock.cc:648] HybridClock initialized: now 1781914339326505 us; error 49 us; skew 500 ppm
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:19.329710 3503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:19.330204 3503 webserver.cc:460] Webserver started at http://127.0.98.1:38769/ using document root <none> and password file <none>
I20260620 00:12:19.330440 3503 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:19.330488 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:19.330587 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:19.331719 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "09875999f010480eb0396f949b3b0199"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "e739be62d4e92e42f5edf7831fc3f734"
server_key_iv: "684aa8909397c7e716863f97a4640f95"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.332056 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09875999f010480eb0396f949b3b0199"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "e739be62d4e92e42f5edf7831fc3f734"
server_key_iv: "684aa8909397c7e716863f97a4640f95"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.333642 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260620 00:12:19.334707 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.335017 3503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:19.335124 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "09875999f010480eb0396f949b3b0199"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "e739be62d4e92e42f5edf7831fc3f734"
server_key_iv: "684aa8909397c7e716863f97a4640f95"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.335258 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:19.345532 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:19.348301 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:19.348435 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:19.348786 3503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:19.349135 3503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:19.349171 3503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.349196 3503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:19.349239 3503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.356209 3503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33641
I20260620 00:12:19.356253 3631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33641 every 8 connection(s)
I20260620 00:12:19.356709 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:19.366889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3503
I20260620 00:12:19.367012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:19.367184 392 external_mini_cluster.cc:1428] Setting key cd139448fec30468dfc7dda935e9dd1e
I20260620 00:12:19.367725 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.357628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57381 (local address 127.0.98.62:44707)
0620 00:12:19.357738 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:19.357742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:19.357836 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:12:19.358466 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:19.358467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:19.358479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:19.358544 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:19.359620 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.360207 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.361433 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.361662 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.361745 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.364352 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:19.364354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:19.364354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:19.364362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:19.365852 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.366360 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.366362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.366362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.366420 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.366751 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.366752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.366752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.366916 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:19.367012 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:19.367551 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 00:12:19.367656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 00:12:19.368039 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.357503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44707 (local address 127.0.98.1:57381)
0620 00:12:19.357703 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:19.357759 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:19.358324 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:19.358631 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:19.358635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:19.359270 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:19.359483 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.359488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.360327 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.360328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.361328 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.361332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.361774 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.361775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.361897 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.362539 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:19.362549 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:19.364232 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:19.365955 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.365969 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.365971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.366277 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.366538 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.366539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.366539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.366670 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.367126 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:19.367135 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:19.367349 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 00:12:19.367504 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 00:12:19.367718 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260620 00:12:19.368705 3632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44707
I20260620 00:12:19.368821 3632 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:19.369046 3632 heartbeater.cc:511] Master 127.0.98.62:44707 requested a full tablet report, sending...
I20260620 00:12:19.369568 3441 ts_manager.cc:194] Registered new tserver with Master: 09875999f010480eb0396f949b3b0199 (127.0.98.1:33641)
I20260620 00:12:19.370363 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57381
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:19.397076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:19.486064 3637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:19.486341 3637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:19.486370 3637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:19.487690 3637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:19.487749 3637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:19.487780 3637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:19.489985 3637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:19.490434 3637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:19.490690 3637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:19.496863 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.496903 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.496863 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.497179 3637 server_base.cc:1034] running on GCE node
I20260620 00:12:19.497507 3637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:19.497761 3637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:19.498982 3637 hybrid_clock.cc:648] HybridClock initialized: now 1781914339498957 us; error 45 us; skew 500 ppm
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:19.501843 3637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:19.502364 3637 webserver.cc:460] Webserver started at http://127.0.98.2:34477/ using document root <none> and password file <none>
I20260620 00:12:19.502640 3637 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:19.502689 3637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:19.502826 3637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:19.504036 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "feda8120ccc040b2845586ba3c922bef"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "5d4e7eccc26389d1daf5b5010b16bdc3"
server_key_iv: "1e9c56f46a8658ba94739fc202de8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.504429 3637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "feda8120ccc040b2845586ba3c922bef"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "5d4e7eccc26389d1daf5b5010b16bdc3"
server_key_iv: "1e9c56f46a8658ba94739fc202de8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.506103 3637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:19.507021 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.507285 3637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:19.507366 3637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "feda8120ccc040b2845586ba3c922bef"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "5d4e7eccc26389d1daf5b5010b16bdc3"
server_key_iv: "1e9c56f46a8658ba94739fc202de8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.507464 3637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:19.540930 3637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:19.544345 3637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:19.544623 3637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:19.544895 3637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:19.545274 3637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:19.545361 3637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.545416 3637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:19.545449 3637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.554236 3637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44263
I20260620 00:12:19.554462 3765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44263 every 8 connection(s)
I20260620 00:12:19.554742 3637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:19.563432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3637
I20260620 00:12:19.563588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:19.563836 392 external_mini_cluster.cc:1428] Setting key 776454e6e849a3fbf0df9f2b213c97e9
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:19.569625 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.555801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40965 (local address 127.0.98.62:44707)
0620 00:12:19.555901 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:19.555903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:19.556107 (+ 204us) server_negotiation.cc:407] Connection header received
0620 00:12:19.556709 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:19.556710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:19.556723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:19.556817 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:19.558161 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.559251 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.561245 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.561491 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.561578 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.564765 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:19.564767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:19.564768 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:19.564778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:19.566818 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.567763 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.567764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.567765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.567899 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.568381 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.568383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.568383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.568795 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:19.568932 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:19.569314 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 00:12:19.569506 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 00:12:19.569885 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.555724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44707 (local address 127.0.98.2:40965)
0620 00:12:19.555923 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:19.555970 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:19.556500 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:19.556937 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:19.556942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:19.557653 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:19.557876 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.557881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.559585 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.559587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.561066 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.561075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.561632 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.561634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.561773 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.562611 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:19.562622 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:19.564634 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:19.567002 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.567011 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.567014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.567403 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.567996 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.567997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.567998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.568202 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.568942 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:19.568952 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:19.569143 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 00:12:19.569273 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 00:12:19.569597 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260620 00:12:19.570720 3766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44707
I20260620 00:12:19.570983 3766 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:19.571219 3766 heartbeater.cc:511] Master 127.0.98.62:44707 requested a full tablet report, sending...
I20260620 00:12:19.571769 3441 ts_manager.cc:194] Registered new tserver with Master: feda8120ccc040b2845586ba3c922bef (127.0.98.2:44263)
I20260620 00:12:19.572391 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40965
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:19.595659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:19.689949 3771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:19.690217 3771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:19.690246 3771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:19.691829 3771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:19.691893 3771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:19.691933 3771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:19.693809 3771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:19.694074 3771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:19.694275 3771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:19.700227 3779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.700227 3776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:19.700332 3777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:19.700582 3771 server_base.cc:1034] running on GCE node
I20260620 00:12:19.701148 3771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:19.701470 3771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:19.702664 3771 hybrid_clock.cc:648] HybridClock initialized: now 1781914339702642 us; error 37 us; skew 500 ppm
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:19.705683 3771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:19.706429 3771 webserver.cc:460] Webserver started at http://127.0.98.3:34117/ using document root <none> and password file <none>
I20260620 00:12:19.706701 3771 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:19.706753 3771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:19.706905 3771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:19.708074 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "494a2898e5724e078915dadf22feded5"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "64c5f8124b40b66f8ce2de4443c89ed3"
server_key_iv: "f947b40f44e7fcad6f0ba6d215a0e44c"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.708457 3771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "494a2898e5724e078915dadf22feded5"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "64c5f8124b40b66f8ce2de4443c89ed3"
server_key_iv: "f947b40f44e7fcad6f0ba6d215a0e44c"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.710036 3771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:19.711054 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.711560 3771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:19.711704 3771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "494a2898e5724e078915dadf22feded5"
format_stamp: "Formatted at 2026-06-20 00:12:19 on dist-test-slave-28bv"
server_key: "64c5f8124b40b66f8ce2de4443c89ed3"
server_key_iv: "f947b40f44e7fcad6f0ba6d215a0e44c"
server_key_version: "encryptionkey@0"
I20260620 00:12:19.711795 3771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:19.731570 3771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:19.735248 3771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:19.735399 3771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:19.735690 3771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:19.736073 3771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:19.736110 3771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.736136 3771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:19.736176 3771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:19.743346 3771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37765
I20260620 00:12:19.743429 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37765 every 8 connection(s)
I20260620 00:12:19.743810 3771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:19.752223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3771
I20260620 00:12:19.752333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:19.752521 392 external_mini_cluster.cc:1428] Setting key 4eefd238616a9c45a6c8f46e69e2b4f9
I20260620 00:12:19.755831 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.744657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53197 (local address 127.0.98.62:44707)
0620 00:12:19.744734 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:19.744736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:19.745018 (+ 282us) server_negotiation.cc:407] Connection header received
0620 00:12:19.745610 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:19.745611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:19.745625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:19.745680 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:19.746797 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.747332 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.749141 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.749413 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.749523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.752314 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:19.752317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:19.752317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:19.752328 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:19.754166 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.754725 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.754727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.754727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.754794 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.755079 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.755080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.755081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.755216 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:19.755295 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:19.755637 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 00:12:19.755733 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:12:19.756084 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.744662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44707 (local address 127.0.98.3:53197)
0620 00:12:19.744837 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:19.744917 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:19.745485 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:19.745778 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:19.745781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:19.746454 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:19.746674 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.746680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.747456 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.747457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.748999 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:19.749003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.749538 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.749540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:19.749827 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.750516 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:19.750526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:19.752148 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:19.754285 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.754296 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.754299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.754636 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.754867 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:19.754868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:19.754869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:19.755000 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:19.755297 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:19.755305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:19.755475 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 00:12:19.755604 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 00:12:19.755792 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260620 00:12:19.756789 3900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44707
I20260620 00:12:19.756899 3900 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:19.757124 3900 heartbeater.cc:511] Master 127.0.98.62:44707 requested a full tablet report, sending...
I20260620 00:12:19.757838 3441 ts_manager.cc:194] Registered new tserver with Master: 494a2898e5724e078915dadf22feded5 (127.0.98.3:37765)
I20260620 00:12:19.758246 3441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53197
I20260620 00:12:19.764863 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914339, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:19.791081 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:19.782992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.0.98.62:44707)
0620 00:12:19.783066 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:19.783068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:19.783105 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:12:19.783181 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:19.783182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:19.783191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:19.783262 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:19.783972 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.784481 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.785322 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:19.785551 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:19.785614 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:19.788054 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:19.788056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:19.788056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:19.788065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:19.789719 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.790156 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.790158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.790158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.790223 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:19.790445 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:19.790445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:19.790446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:19.790624 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:19.790708 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:19.790930 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 00:12:19.791007 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 00:12:19.791498 3441 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34786
I20260620 00:12:19.792052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3503
I20260620 00:12:19.797169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3637
I20260620 00:12:19.802481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3771
I20260620 00:12:19.807777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3409
2026-06-20T00:12:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1965 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3912](info): set up 2 sockets
Jun 20 00:12:19 dist-test-slave-28bv krb5kdc[3912](info): commencing operation
krb5kdc: starting...
W20260620 00:12:20.896778 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:20 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:21.042258 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:21.143239 3925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:21.143572 3925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:21.143620 3925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:21.145360 3925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:21.145496 3925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:21.145527 3925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:21.145584 3925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:21.145615 3925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:21.149550 3925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:21.150167 3925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:21.150501 3925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:21.158231 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.158231 3933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.158674 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:21.158816 3925 server_base.cc:1034] running on GCE node
I20260620 00:12:21.159116 3925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:21.159514 3925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:21.160699 3925 hybrid_clock.cc:648] HybridClock initialized: now 1781914341160673 us; error 33 us; skew 500 ppm
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:21.164239 3925 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:21.164840 3925 webserver.cc:460] Webserver started at http://127.0.98.62:33255/ using document root <none> and password file <none>
I20260620 00:12:21.165081 3925 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:21.165268 3925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:21.165393 3925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:21.166582 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "ec81628e7d234100a333d032052affe7"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "f7f08b7a8a8c85817f9f825f30be0797"
server_key_iv: "c3bfd2cdfe08335ea51a14c26cc3d4c2"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.167083 3925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec81628e7d234100a333d032052affe7"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "f7f08b7a8a8c85817f9f825f30be0797"
server_key_iv: "c3bfd2cdfe08335ea51a14c26cc3d4c2"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.169020 3925 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 00:12:21.170538 3940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.170850 3925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:21.171015 3925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "ec81628e7d234100a333d032052affe7"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "f7f08b7a8a8c85817f9f825f30be0797"
server_key_iv: "c3bfd2cdfe08335ea51a14c26cc3d4c2"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.171123 3925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:21.184897 3925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:21.187954 3925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:21.188160 3925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:21.193105 3925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40033
I20260620 00:12:21.193164 3992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40033 every 8 connection(s)
I20260620 00:12:21.193583 3925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:21.194643 3993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:21.198871 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7: Bootstrap starting.
I20260620 00:12:21.199404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 3925
I20260620 00:12:21.199584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:21.199792 392 external_mini_cluster.cc:1428] Setting key dddaa150a0a6afab55b5a8751a942dbd
I20260620 00:12:21.200048 3993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:21.200524 3993 log.cc:824] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:21.201507 3993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7: No bootstrap required, opened a new log
I20260620 00:12:21.204259 3993 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 00:12:21.204480 3993 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:21.204526 3993 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec81628e7d234100a333d032052affe7, State: Initialized, Role: FOLLOWER
I20260620 00:12:21.204704 3993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 00:12:21.204795 3993 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:21.204839 3993 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:21.204893 3993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:21.205790 3993 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 00:12:21.205960 3993 leader_election.cc:302] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec81628e7d234100a333d032052affe7; no voters:
I20260620 00:12:21.206194 3993 leader_election.cc:288] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:21.206260 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:21.206517 3993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:21.206869 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [term 1 LEADER]: Becoming Leader. State: Replica: ec81628e7d234100a333d032052affe7, State: Running, Role: LEADER
I20260620 00:12:21.206990 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260620 00:12:21.207626 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec81628e7d234100a333d032052affe7. Latest consensus state: current_term: 1 leader_uuid: "ec81628e7d234100a333d032052affe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260620 00:12:21.207718 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:21.207796 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec81628e7d234100a333d032052affe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec81628e7d234100a333d032052affe7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260620 00:12:21.207854 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:21.207933 4007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:21.208501 4007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:21.210911 4007 catalog_manager.cc:1347] Generated new cluster ID: 598596337c8146048f6c0b6be4c1e774
I20260620 00:12:21.210974 4007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:21.216022 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.200451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.98.62:40033)
0620 00:12:21.200685 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.200697 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.200735 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:12:21.201985 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.201995 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.202060 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.202211 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.203327 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.204335 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.205481 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.206065 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.206222 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.209800 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.209807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.209807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.209851 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.213748 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.214363 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.214367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.214367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.214500 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.214821 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.214822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.214822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.215187 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.215350 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.215613 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.215774 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":134,"threads_started":1}
I20260620 00:12:21.225677 4007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:21.226544 4007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:21.233431 4007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec81628e7d234100a333d032052affe7: Generated new TSK 0
I20260620 00:12:21.233697 4007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:21.277647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:21.369282 4019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:21.369542 4019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:21.369571 4019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:21.370954 4019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:21.371021 4019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:21.371050 4019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:21.372903 4019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:21.373206 4019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:21.373490 4019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:21.379312 4024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.379258 4025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.379372 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:21.379633 4019 server_base.cc:1034] running on GCE node
I20260620 00:12:21.379905 4019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:21.380143 4019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:21.381306 4019 hybrid_clock.cc:648] HybridClock initialized: now 1781914341381261 us; error 66 us; skew 500 ppm
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:21.384349 4019 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:21.384967 4019 webserver.cc:460] Webserver started at http://127.0.98.1:41325/ using document root <none> and password file <none>
I20260620 00:12:21.385304 4019 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:21.385388 4019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:21.385548 4019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:21.386713 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a2f56b20d3c4467a240a049064507b6"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "0ad0c101270348a9a071328ec27e33b0"
server_key_iv: "1b6b6c555b9db76e910f5d8f1091b736"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.387213 4019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2f56b20d3c4467a240a049064507b6"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "0ad0c101270348a9a071328ec27e33b0"
server_key_iv: "1b6b6c555b9db76e910f5d8f1091b736"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.389086 4019 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:12:21.389994 4034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.390192 4019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:21.390309 4019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "9a2f56b20d3c4467a240a049064507b6"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "0ad0c101270348a9a071328ec27e33b0"
server_key_iv: "1b6b6c555b9db76e910f5d8f1091b736"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.390406 4019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:21.432471 4019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:21.435798 4019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:21.436003 4019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:21.436305 4019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:21.436700 4019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:21.436764 4019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.436810 4019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:21.436837 4019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.444068 4019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40653
I20260620 00:12:21.444124 4147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40653 every 8 connection(s)
I20260620 00:12:21.444629 4019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:21.444830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4019
I20260620 00:12:21.445019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:21.445254 392 external_mini_cluster.cc:1428] Setting key 20faeb2b0d2962838a5b18a4e854199a
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:12:21.458715 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.445490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32873 (local address 127.0.98.62:40033)
0620 00:12:21.445557 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.445561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.445866 (+ 305us) server_negotiation.cc:407] Connection header received
0620 00:12:21.447079 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.447080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.447094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.447145 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.448410 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.449099 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.450900 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.451189 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.451302 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.454748 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.454750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.454750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.454760 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.456719 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.457369 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.457370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.457370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.457439 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.457857 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.457858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.457859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.458030 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.458117 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.458503 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.458612 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:12:21.459060 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.445498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.1:32873)
0620 00:12:21.445685 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:21.445740 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:21.446912 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:21.447245 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:21.447250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:21.447573 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:21.448162 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.448177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.449412 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.449419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.450586 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.450591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.451324 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.451325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.451582 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.452373 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:21.452384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:21.454437 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:21.456881 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.456892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.456895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.457213 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.457551 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.457552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.457552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.457764 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.458108 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:21.458115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:21.458348 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 00:12:21.458481 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 00:12:21.458674 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260620 00:12:21.460033 4148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 00:12:21.460147 4148 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:21.460423 4148 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 00:12:21.461061 3956 ts_manager.cc:194] Registered new tserver with Master: 9a2f56b20d3c4467a240a049064507b6 (127.0.98.1:40653)
I20260620 00:12:21.461704 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:21.476943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:21.562875 4153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:21.563156 4153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:21.563198 4153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:21.564617 4153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:21.564697 4153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:21.564744 4153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:21.566455 4153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:21.566771 4153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:21.567024 4153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:21.572618 4161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.572634 4158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.572639 4159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:21.572666 4153 server_base.cc:1034] running on GCE node
I20260620 00:12:21.573170 4153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:21.573442 4153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:21.574626 4153 hybrid_clock.cc:648] HybridClock initialized: now 1781914341574613 us; error 35 us; skew 500 ppm
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:21.577505 4153 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:21.577996 4153 webserver.cc:460] Webserver started at http://127.0.98.2:46387/ using document root <none> and password file <none>
I20260620 00:12:21.578246 4153 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:21.578315 4153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:21.578469 4153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:21.579553 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "df5b53c6257f4dbe9da62bc06f07cc1a"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "3543ab042a20979121db612c2e24f56b"
server_key_iv: "d37395c0e61937dcc42914aaf8633e94"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.579986 4153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "df5b53c6257f4dbe9da62bc06f07cc1a"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "3543ab042a20979121db612c2e24f56b"
server_key_iv: "d37395c0e61937dcc42914aaf8633e94"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.581660 4153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:21.582674 4168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.582891 4153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:21.582994 4153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "df5b53c6257f4dbe9da62bc06f07cc1a"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "3543ab042a20979121db612c2e24f56b"
server_key_iv: "d37395c0e61937dcc42914aaf8633e94"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.583081 4153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:21.611935 4153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:21.614821 4153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:21.614988 4153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:21.615367 4153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:21.615888 4153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:21.615942 4153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.615969 4153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:21.615984 4153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.623072 4153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33605
I20260620 00:12:21.623155 4281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33605 every 8 connection(s)
I20260620 00:12:21.623548 4153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:21.633167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4153
I20260620 00:12:21.633304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:21.633494 392 external_mini_cluster.cc:1428] Setting key 1f69812e000abdbb0bf14b06040edf41
I20260620 00:12:21.633812 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.624575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52519 (local address 127.0.98.62:40033)
0620 00:12:21.624624 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.624626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.624761 (+ 135us) server_negotiation.cc:407] Connection header received
0620 00:12:21.625765 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.625767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.625777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.625842 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.626458 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.626971 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.627982 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.628168 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.628273 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.630616 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.630617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.630617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.630625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.632194 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.632652 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.632653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.632654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.632714 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.633059 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.633059 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.633060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.633166 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.633215 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.633607 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.633711 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:12:21.634011 4284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.624436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.2:52519)
0620 00:12:21.624623 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:21.624671 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:21.625650 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:21.625902 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:21.625905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:21.626174 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:21.626380 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.626384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.627057 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.627058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.627899 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.627901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.628294 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.628295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.628396 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.629040 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:21.629048 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:21.630502 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:21.632311 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.632317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.632319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.632542 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.632811 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.632812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.632812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.632924 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.633231 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:21.633237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:21.633424 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 00:12:21.633539 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:12:21.633728 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":132,"threads_started":1}
I20260620 00:12:21.634460 4282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 00:12:21.634565 4282 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:21.634783 4282 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 00:12:21.635142 3956 ts_manager.cc:194] Registered new tserver with Master: df5b53c6257f4dbe9da62bc06f07cc1a (127.0.98.2:33605)
I20260620 00:12:21.635591 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52519
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:21.660362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:21.742976 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:21.743232 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:21.743259 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:21.744606 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:21.744652 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:21.744681 4287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:21.746243 4287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:21.746449 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:21.746640 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:21.751950 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.751950 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:21.751959 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:21.752192 4287 server_base.cc:1034] running on GCE node
I20260620 00:12:21.752460 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:21.752830 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:21.753983 4287 hybrid_clock.cc:648] HybridClock initialized: now 1781914341753935 us; error 69 us; skew 500 ppm
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:21.756877 4287 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:21.757350 4287 webserver.cc:460] Webserver started at http://127.0.98.3:40143/ using document root <none> and password file <none>
I20260620 00:12:21.757635 4287 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:21.757709 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:21.757863 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:21.758948 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b771b237c3a46f1ac421617aefd2be8"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "d58fe0e6bbef290f7765359372bd2f8a"
server_key_iv: "69220d38bf795e32b439fc94a95a365c"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.759310 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b771b237c3a46f1ac421617aefd2be8"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "d58fe0e6bbef290f7765359372bd2f8a"
server_key_iv: "69220d38bf795e32b439fc94a95a365c"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.760910 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:21.761775 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.761976 4287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:21.762086 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "9b771b237c3a46f1ac421617aefd2be8"
format_stamp: "Formatted at 2026-06-20 00:12:21 on dist-test-slave-28bv"
server_key: "d58fe0e6bbef290f7765359372bd2f8a"
server_key_iv: "69220d38bf795e32b439fc94a95a365c"
server_key_version: "encryptionkey@0"
I20260620 00:12:21.762168 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:21.771996 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:21.774591 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:21.774780 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:21.775049 4287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:21.775416 4287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:21.775452 4287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.775477 4287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:21.775490 4287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:21.781540 4287 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44535
I20260620 00:12:21.781602 4415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44535 every 8 connection(s)
I20260620 00:12:21.781921 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:21.785319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4287
I20260620 00:12:21.785411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:21.785630 392 external_mini_cluster.cc:1428] Setting key ffa5cacc91c503255d4f1fb9589705a0
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:21.792057 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.782753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49527 (local address 127.0.98.62:40033)
0620 00:12:21.782803 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.782806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.783055 (+ 249us) server_negotiation.cc:407] Connection header received
0620 00:12:21.784061 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.784062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.784073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.784132 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.784856 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.785545 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.786845 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.787040 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.787151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.789424 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.789426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.789426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.789433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.790843 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.791242 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.791242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.791243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.791292 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.791505 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.791506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.791506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.791606 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.791657 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.791930 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.791989 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:12:21.792222 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.782745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40033 (local address 127.0.98.3:49527)
0620 00:12:21.782934 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:21.782972 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:21.783940 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:21.784197 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:21.784201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:21.784477 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:21.784750 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.784754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.785642 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.785644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.786729 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:21.786732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.787155 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.787155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:21.787256 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.787866 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:21.787876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:21.789332 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:21.790931 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.790937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.790938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.791174 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.791352 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:21.791353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:21.791353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:21.791450 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:21.791662 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:21.791666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:21.791781 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 00:12:21.791887 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:12:21.792009 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260620 00:12:21.792762 4416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40033
I20260620 00:12:21.792878 4416 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:21.793030 4416 heartbeater.cc:511] Master 127.0.98.62:40033 requested a full tablet report, sending...
I20260620 00:12:21.793524 3956 ts_manager.cc:194] Registered new tserver with Master: 9b771b237c3a46f1ac421617aefd2be8 (127.0.98.3:44535)
I20260620 00:12:21.793892 3956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49527
I20260620 00:12:21.797375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:21.821317 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.812856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.0.98.1:40653)
0620 00:12:21.813028 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.813034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.813054 (+ 20us) server_negotiation.cc:407] Connection header received
0620 00:12:21.813082 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.813084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.813098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.813165 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.813813 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.814550 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.815380 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.815720 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.815786 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.818375 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.818384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.818385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.818392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.819967 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.820348 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.820351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.820352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.820414 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.820682 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.820683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.820683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.820849 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.820972 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.821151 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.821232 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[3912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:21.830825 3996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:21.823077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.0.98.62:40033)
0620 00:12:21.823151 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:21.823153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:21.823166 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:12:21.823203 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:21.823204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:21.823221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:21.823301 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:21.823858 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.824357 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.825208 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:21.825400 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:21.825506 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:21.827941 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:21.827943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:21.827943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:21.827951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:21.829516 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.829911 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.829913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.829913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.829977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:21.830201 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:21.830202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:21.830202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:21.830376 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:21.830495 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:21.830615 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:12:21.830715 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 00:12:21.831111 3956 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698
I20260620 00:12:21.831596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4019
I20260620 00:12:21.836265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4153
I20260620 00:12:21.840931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4287
I20260620 00:12:21.845335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 3925
2026-06-20T00:12:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2036 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[4428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[4428](info): set up 2 sockets
Jun 20 00:12:21 dist-test-slave-28bv krb5kdc[4428](info): commencing operation
krb5kdc: starting...
W20260620 00:12:22.932641 392 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:23.056368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:23.136124 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:23.136341 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:23.136363 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:23.137586 4441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:23.137637 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:23.137652 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:23.137665 4441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:23.137677 4441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:23.139227 4441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:23.139496 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:23.139737 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:23.144830 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.144838 4441 server_base.cc:1034] running on GCE node
W20260620 00:12:23.144809 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.144821 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.145293 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:23.145630 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:23.146796 4441 hybrid_clock.cc:648] HybridClock initialized: now 1781914343146791 us; error 38 us; skew 500 ppm
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:23.154568 4441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:23.154975 4441 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:23.155049 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:23.155221 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:23.156371 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "c5a66b346db847559db7f9f9b5a34dbb"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "282d89ac355984b55e651a21ba927bda"
server_key_iv: "a7d86be57373a06ac87fe875e3bba53e"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.156772 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5a66b346db847559db7f9f9b5a34dbb"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "282d89ac355984b55e651a21ba927bda"
server_key_iv: "a7d86be57373a06ac87fe875e3bba53e"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.158159 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:23.158921 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.159159 4441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:23.159283 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "c5a66b346db847559db7f9f9b5a34dbb"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "282d89ac355984b55e651a21ba927bda"
server_key_iv: "a7d86be57373a06ac87fe875e3bba53e"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.159379 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:23.167949 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:23.170125 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:23.170313 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:23.174582 4441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34613
I20260620 00:12:23.174604 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34613 every 8 connection(s)
I20260620 00:12:23.174911 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:23.175859 4508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:23.178352 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: Bootstrap starting.
I20260620 00:12:23.178993 4508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:23.179311 4508 log.cc:824] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:23.180052 4508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: No bootstrap required, opened a new log
I20260620 00:12:23.181105 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4441
I20260620 00:12:23.181186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:23.181386 392 external_mini_cluster.cc:1428] Setting key 0207a3861f73ae9f744f300b90b851f0
I20260620 00:12:23.182447 4508 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } }
I20260620 00:12:23.182642 4508 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:23.182678 4508 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a66b346db847559db7f9f9b5a34dbb, State: Initialized, Role: FOLLOWER
I20260620 00:12:23.182816 4508 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } }
I20260620 00:12:23.182904 4508 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:23.182934 4508 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:23.183024 4508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:23.183666 4508 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } }
I20260620 00:12:23.183781 4508 leader_election.cc:302] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5a66b346db847559db7f9f9b5a34dbb; no voters:
I20260620 00:12:23.183938 4508 leader_election.cc:288] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:23.184007 4513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:23.184180 4513 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [term 1 LEADER]: Becoming Leader. State: Replica: c5a66b346db847559db7f9f9b5a34dbb, State: Running, Role: LEADER
I20260620 00:12:23.184183 4508 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:23.184955 4513 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } }
I20260620 00:12:23.185617 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5a66b346db847559db7f9f9b5a34dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } } }
I20260620 00:12:23.185757 4514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5a66b346db847559db7f9f9b5a34dbb. Latest consensus state: current_term: 1 leader_uuid: "c5a66b346db847559db7f9f9b5a34dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5a66b346db847559db7f9f9b5a34dbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34613 } } }
I20260620 00:12:23.186050 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:23.186064 4514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb [sys.catalog]: This master's current role is: LEADER
W20260620 00:12:23.186224 4527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:23.186271 4527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:23.186359 4529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:23.186564 4529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:23.188287 4529 catalog_manager.cc:1347] Generated new cluster ID: 2eb7268e51d9407ebb5b9b7c1276cf7b
I20260620 00:12:23.188344 4529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:23.193809 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.182055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.0.98.62:34613)
0620 00:12:23.182499 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.182509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.182538 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:12:23.182651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.182659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.182720 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.182867 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:23.183633 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.185004 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.186119 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.186763 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.186865 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.189892 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:23.189896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:23.189896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:23.189909 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:23.191916 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.192431 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.192434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.192434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.192594 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.192824 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.192824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.192825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.193156 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:23.193302 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.193443 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.193588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260620 00:12:23.207217 4529 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:23.207803 4529 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:23.215809 4529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: Generated new TSK 0
I20260620 00:12:23.216058 4529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:23.244858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:23.329335 4534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:23.329548 4534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:23.329576 4534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:23.330843 4534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:23.330890 4534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:23.330919 4534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:23.332567 4534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:23.332842 4534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:23.333091 4534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:23.338191 4540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.338244 4539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.338297 4542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.338430 4534 server_base.cc:1034] running on GCE node
I20260620 00:12:23.338699 4534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:23.338969 4534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:23.340173 4534 hybrid_clock.cc:648] HybridClock initialized: now 1781914343340148 us; error 70 us; skew 500 ppm
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:23.342828 4534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:23.343233 4534 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:23.343312 4534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:23.343463 4534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:23.344667 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "08dcd1e7651844b8bc3042932d347c92"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "1bc828f2e31b4add5266c2ed133f563c"
server_key_iv: "3368598545a581258a202af8dbdd51d7"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.345034 4534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08dcd1e7651844b8bc3042932d347c92"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "1bc828f2e31b4add5266c2ed133f563c"
server_key_iv: "3368598545a581258a202af8dbdd51d7"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.346449 4534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:23.347268 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.347431 4534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:23.347540 4534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "08dcd1e7651844b8bc3042932d347c92"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "1bc828f2e31b4add5266c2ed133f563c"
server_key_iv: "3368598545a581258a202af8dbdd51d7"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.347638 4534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:23.356400 4534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:23.358786 4534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:23.358979 4534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:23.359232 4534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:23.359562 4534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:23.359643 4534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.359690 4534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:23.359715 4534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.366022 4534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34159
I20260620 00:12:23.366060 4661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34159 every 8 connection(s)
I20260620 00:12:23.366366 4534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:23.369900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4534
I20260620 00:12:23.370023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:23.370218 392 external_mini_cluster.cc:1428] Setting key 31e202d8c93160f7784ce8c739157c16
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:23.377355 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.367261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51987 (local address 127.0.98.62:34613)
0620 00:12:23.367332 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.367333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.367539 (+ 206us) server_negotiation.cc:407] Connection header received
0620 00:12:23.367995 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.367996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.368011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.368059 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:23.369090 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.369621 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.370923 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.371231 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.371290 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.373900 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:23.373901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:23.373902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:23.373911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:23.375521 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.376117 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.376118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.376118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.376177 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.376483 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.376483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.376483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.376659 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:23.376714 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.377110 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.377226 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:12:23.377930 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.367192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34613 (local address 127.0.98.1:51987)
0620 00:12:23.367400 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.367437 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.367914 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.368126 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.368129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.368397 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:23.368996 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.369002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.369749 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.369751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.370811 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.370814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.371323 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.371324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.371481 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.372025 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:23.372035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:23.373781 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:23.375620 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.375631 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.375634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.376010 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.376247 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.376248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.376249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.376408 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.376728 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:23.376736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:23.376926 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.377479 (+ 553us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.377673 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":88,"threads_started":1}
I20260620 00:12:23.378634 4662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34613
I20260620 00:12:23.378742 4662 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:23.378942 4662 heartbeater.cc:511] Master 127.0.98.62:34613 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:12:23.379462 4472 ts_manager.cc:194] Registered new tserver with Master: 08dcd1e7651844b8bc3042932d347c92 (127.0.98.1:34159)
I20260620 00:12:23.380100 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:23.399312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:23.483491 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:23.483872 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:23.483919 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:23.485458 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:23.485539 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:23.485592 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:23.487413 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:23.487725 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:23.487978 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:23.493337 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.493454 4667 server_base.cc:1034] running on GCE node
W20260620 00:12:23.493332 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.493333 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.493767 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:23.494056 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:23.495256 4667 hybrid_clock.cc:648] HybridClock initialized: now 1781914343495243 us; error 36 us; skew 500 ppm
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:23.498131 4667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:23.498540 4667 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:23.498595 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:23.498697 4667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:23.499806 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "d61d9a305e944746a1a2d267e043628c"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ef24df52b122a382ba1b3b34d8a6f29d"
server_key_iv: "5aa9752d4640c21d79e46e0dedc09572"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.500123 4667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d61d9a305e944746a1a2d267e043628c"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ef24df52b122a382ba1b3b34d8a6f29d"
server_key_iv: "5aa9752d4640c21d79e46e0dedc09572"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.501600 4667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:23.502394 4681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.502624 4667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:23.502712 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "d61d9a305e944746a1a2d267e043628c"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ef24df52b122a382ba1b3b34d8a6f29d"
server_key_iv: "5aa9752d4640c21d79e46e0dedc09572"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.502774 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:23.514849 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:23.517163 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:23.517288 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:23.517488 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:23.517844 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:23.517880 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.517903 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:23.517952 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.524287 4667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35427
I20260620 00:12:23.524353 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35427 every 8 connection(s)
I20260620 00:12:23.524710 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:23.534636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4667
I20260620 00:12:23.534756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:23.534987 392 external_mini_cluster.cc:1428] Setting key c50ef5789b0889a89031111ef28cd8b7
I20260620 00:12:23.535156 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.525626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48105 (local address 127.0.98.62:34613)
0620 00:12:23.525706 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.525708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.525931 (+ 223us) server_negotiation.cc:407] Connection header received
0620 00:12:23.526517 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.526519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.526530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.526590 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:23.527689 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.528223 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.529475 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.529676 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.529750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.532325 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:23.532327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:23.532327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:23.532334 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:23.533643 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.534031 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.534031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.534032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.534080 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.534343 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.534345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.534345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.534550 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:23.534637 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.534966 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.535061 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:12:23.535386 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.525584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34613 (local address 127.0.98.2:48105)
0620 00:12:23.525796 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.525837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.526402 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.526658 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.526662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.526971 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:23.527553 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.527560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.528370 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.528371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.529348 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.529351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.529803 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.529804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.529927 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.530638 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:23.530647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:23.532220 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:23.533724 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.533731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.533733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.533965 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.534139 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.534140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.534141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.534253 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.534641 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:23.534647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:23.534821 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.534926 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.535100 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":105,"threads_started":1}
I20260620 00:12:23.535970 4795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34613
I20260620 00:12:23.536098 4795 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:23.536269 4795 heartbeater.cc:511] Master 127.0.98.62:34613 requested a full tablet report, sending...
I20260620 00:12:23.536695 4472 ts_manager.cc:194] Registered new tserver with Master: d61d9a305e944746a1a2d267e043628c (127.0.98.2:35427)
I20260620 00:12:23.537073 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48105
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:23.563473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:23.645359 4800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:23.645581 4800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:23.645605 4800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:23.646733 4800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:23.646775 4800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:23.646801 4800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:23.648272 4800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:23.648473 4800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:23.648767 4800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:23.653817 4805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.653823 4808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:23.653873 4806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:23.653949 4800 server_base.cc:1034] running on GCE node
I20260620 00:12:23.654258 4800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:23.654467 4800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:23.655629 4800 hybrid_clock.cc:648] HybridClock initialized: now 1781914343655618 us; error 34 us; skew 500 ppm
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:23.658310 4800 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:23.658594 4800 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:23.658636 4800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:23.658728 4800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:23.659740 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba4c05c5b60540649aa5935f185d7675"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ac9770e8339d28e2c09edd0925744a86"
server_key_iv: "dd081bf0415e9388eff0b312b540d46a"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.660032 4800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba4c05c5b60540649aa5935f185d7675"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ac9770e8339d28e2c09edd0925744a86"
server_key_iv: "dd081bf0415e9388eff0b312b540d46a"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.661371 4800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:23.662127 4814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.662297 4800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:23.662395 4800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "ba4c05c5b60540649aa5935f185d7675"
format_stamp: "Formatted at 2026-06-20 00:12:23 on dist-test-slave-28bv"
server_key: "ac9770e8339d28e2c09edd0925744a86"
server_key_iv: "dd081bf0415e9388eff0b312b540d46a"
server_key_version: "encryptionkey@0"
I20260620 00:12:23.662477 4800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:23.682190 4800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:23.685271 4800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:23.685436 4800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:23.685705 4800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:23.686101 4800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:23.686149 4800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.686174 4800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:23.686234 4800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:23.692077 4800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42803
I20260620 00:12:23.692147 4927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42803 every 8 connection(s)
I20260620 00:12:23.692459 4800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:23 dist-test-slave-28bv krb5kdc[4428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:23.698794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4800
I20260620 00:12:23.698877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:23.699076 392 external_mini_cluster.cc:1428] Setting key 86bd5ac219b702c8eab4f7230f5e60ac
I20260620 00:12:23.702176 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.693293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33935 (local address 127.0.98.62:34613)
0620 00:12:23.693363 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.693365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.693576 (+ 211us) server_negotiation.cc:407] Connection header received
0620 00:12:23.694070 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.694070 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.694081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.694160 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:23.695056 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.695588 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.696611 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.696815 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.696873 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.699230 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:23.699231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:23.699232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:23.699239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:23.700786 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.701187 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.701188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.701188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.701249 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.701541 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.701541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.701542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.701656 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:23.701718 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.702037 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.702106 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:12:23.702518 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.693272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34613 (local address 127.0.98.3:33935)
0620 00:12:23.693451 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.693489 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.693975 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.694192 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.694195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.694472 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:23.694977 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.694981 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.695678 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.695679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.696500 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.696502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.696892 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.696893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.697000 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.697519 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:23.697527 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:23.699077 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:23.700857 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.700864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.700867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.701094 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.701339 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:23.701340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:23.701341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:23.701456 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:23.701752 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:23.701758 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:23.701894 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.702003 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.702193 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260620 00:12:23.703055 4928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34613
I20260620 00:12:23.703190 4928 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:23.703364 4928 heartbeater.cc:511] Master 127.0.98.62:34613 requested a full tablet report, sending...
I20260620 00:12:23.703756 4472 ts_manager.cc:194] Registered new tserver with Master: ba4c05c5b60540649aa5935f185d7675 (127.0.98.3:42803)
I20260620 00:12:23.704149 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33935
I20260620 00:12:23.711225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:12:23.719064 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.712608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.0.98.62:34613)
0620 00:12:23.712676 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.712678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.712844 (+ 166us) server_negotiation.cc:407] Connection header received
0620 00:12:23.712883 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.712884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.712893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.712943 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:23.713603 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.714126 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.714889 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.715070 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.715129 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.716716 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:23.716717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:23.716717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:23.716725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:23.718178 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.718465 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.718465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.718466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.718513 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:23.718653 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:23.718653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:23.718654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:23.718786 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:23.718856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.718943 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.718999 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:12:23.720649 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:23.722224 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:23.728533 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42803 (local address 127.0.0.1:52698)
0620 00:12:23.724675 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.724737 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.724807 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.725040 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.725044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.725046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:23.725115 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.725117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726187 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.726188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.727249 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.727251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728183 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.728276 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.728294 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.728397 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.728454 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
I20260620 00:12:23.728569 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.0.98.3:42803)
0620 00:12:23.724859 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.724864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.724875 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:23.724917 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.724919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.724936 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.725021 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:23.725196 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726077 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.727347 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728091 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.728242 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.728416 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.728454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":96,"threads_started":1}
I20260620 00:12:23.729055 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35427 (local address 127.0.0.1:33578)
0620 00:12:23.725009 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.725019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.725066 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.725461 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.725461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.725463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:23.725507 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.725509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726798 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.726799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.727899 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.727901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728818 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.728886 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.728889 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.728950 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.728989 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":27,"thread_start_us":119,"threads_started":1}
I20260620 00:12:23.729089 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.0.98.2:35427)
0620 00:12:23.725218 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.725223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.725232 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:23.725272 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.725274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.725289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.725382 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:23.725587 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726677 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.727988 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728753 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728819 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.728870 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.728963 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.729002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260620 00:12:23.729321 4862 tablet_service.cc:1467] Processing CreateTablet for tablet 44abfbc1f056401b8e57c651bd971237 (DEFAULT_TABLE table=test-table [id=f9d0df9eb4d145cf8b79b990d1e05456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:23.729737 4862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44abfbc1f056401b8e57c651bd971237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:23.729784 4729 tablet_service.cc:1467] Processing CreateTablet for tablet 44abfbc1f056401b8e57c651bd971237 (DEFAULT_TABLE table=test-table [id=f9d0df9eb4d145cf8b79b990d1e05456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:23.729888 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.98.1:34159)
0620 00:12:23.725035 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.725040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.725200 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:12:23.725282 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.725284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.725298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.725374 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:23.725587 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726706 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.727795 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728500 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.729716 (+ 1124us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.729743 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.729793 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260620 00:12:23.730057 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.724725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34159 (local address 127.0.0.1:38666)
0620 00:12:23.725120 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.725129 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.725176 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.725461 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.725461 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.725466 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:23.725507 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.725510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.726805 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.726805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.727712 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.727714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.728986 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.728986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.729057 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.729063 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.729465 (+ 402us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.729502 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":30,"thread_start_us":52,"threads_started":1}
I20260620 00:12:23.730101 4729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44abfbc1f056401b8e57c651bd971237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:23.733280 4946 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Bootstrap starting.
I20260620 00:12:23.733217 4596 tablet_service.cc:1467] Processing CreateTablet for tablet 44abfbc1f056401b8e57c651bd971237 (DEFAULT_TABLE table=test-table [id=f9d0df9eb4d145cf8b79b990d1e05456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:23.733557 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44abfbc1f056401b8e57c651bd971237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:23.733954 4946 tablet_bootstrap.cc:654] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:23.734555 4946 log.cc:824] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:23.734930 4945 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Bootstrap starting.
I20260620 00:12:23.735293 4946 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: No bootstrap required, opened a new log
I20260620 00:12:23.735343 4946 ts_tablet_manager.cc:1397] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:23.735572 4945 tablet_bootstrap.cc:654] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:23.736160 4945 log.cc:824] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:23.736958 4945 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: No bootstrap required, opened a new log
I20260620 00:12:23.737030 4945 ts_tablet_manager.cc:1397] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:23.737190 4946 raft_consensus.cc:348] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.737201 4949 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Bootstrap starting.
I20260620 00:12:23.737365 4946 raft_consensus.cc:374] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:23.737407 4946 raft_consensus.cc:729] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4c05c5b60540649aa5935f185d7675, State: Initialized, Role: FOLLOWER
I20260620 00:12:23.737545 4946 consensus_queue.cc:260] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.737840 4946 ts_tablet_manager.cc:1428] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:12:23.737855 4928 heartbeater.cc:503] Master 127.0.98.62:34613 was elected leader, sending a full tablet report...
I20260620 00:12:23.737965 4949 tablet_bootstrap.cc:654] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:23.738523 4949 log.cc:824] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:23.739053 4945 raft_consensus.cc:348] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.739250 4945 raft_consensus.cc:374] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:23.739308 4945 raft_consensus.cc:729] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61d9a305e944746a1a2d267e043628c, State: Initialized, Role: FOLLOWER
I20260620 00:12:23.739327 4949 tablet_bootstrap.cc:492] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: No bootstrap required, opened a new log
I20260620 00:12:23.739385 4949 ts_tablet_manager.cc:1397] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:12:23.739427 4945 consensus_queue.cc:260] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.739701 4945 ts_tablet_manager.cc:1428] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:23.739773 4795 heartbeater.cc:503] Master 127.0.98.62:34613 was elected leader, sending a full tablet report...
I20260620 00:12:23.741375 4949 raft_consensus.cc:348] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.741530 4949 raft_consensus.cc:374] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:23.741577 4949 raft_consensus.cc:729] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dcd1e7651844b8bc3042932d347c92, State: Initialized, Role: FOLLOWER
I20260620 00:12:23.741695 4949 consensus_queue.cc:260] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.741989 4949 ts_tablet_manager.cc:1428] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:23.742045 4662 heartbeater.cc:503] Master 127.0.98.62:34613 was elected leader, sending a full tablet report...
W20260620 00:12:23.775296 4796 tablet.cc:2369] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:12:23.867092 4663 tablet.cc:2369] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:23.873276 4952 raft_consensus.cc:482] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:23.873401 4952 raft_consensus.cc:504] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.873770 4952 leader_election.cc:288] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba4c05c5b60540649aa5935f185d7675 (127.0.98.3:42803), 08dcd1e7651844b8bc3042932d347c92 (127.0.98.1:34159)
I20260620 00:12:23.876776 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.874025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34635 (local address 127.0.98.3:42803)
0620 00:12:23.874073 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.874076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.874088 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:23.874239 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.874240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.874257 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.874325 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:23.874485 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.874963 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.875987 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.876366 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.876451 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.876461 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.876663 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.876700 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:12:23.876776 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.873962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42803 (local address 127.0.98.2:34635)
0620 00:12:23.874040 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.874053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.874098 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.874329 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.874329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.874332 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:23.874402 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.874404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.875055 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.875062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.875882 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.875885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.876461 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.876462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.876594 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.876597 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.876649 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.876698 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 00:12:23.876866 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.874071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55369 (local address 127.0.98.1:34159)
0620 00:12:23.874112 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.874114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.874303 (+ 189us) server_negotiation.cc:407] Connection header received
0620 00:12:23.874379 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.874380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.874393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.874455 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:23.874640 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.875113 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.876078 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.876454 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.876541 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.876551 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.876732 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.876769 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:23.877111 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.873998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34159 (local address 127.0.98.2:55369)
0620 00:12:23.874220 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.874228 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:23.874309 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:23.874472 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:23.874473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:23.874475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:23.874550 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.874552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.875202 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.875203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.875981 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:23.875983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.876565 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.876566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:23.876642 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.876646 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:12:23.876724 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:12:23.876764 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":129,"threads_started":1}
I20260620 00:12:23.877177 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44abfbc1f056401b8e57c651bd971237" candidate_uuid: "d61d9a305e944746a1a2d267e043628c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4c05c5b60540649aa5935f185d7675" is_pre_election: true
I20260620 00:12:23.877381 4882 raft_consensus.cc:2454] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61d9a305e944746a1a2d267e043628c in term 0.
I20260620 00:12:23.877461 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44abfbc1f056401b8e57c651bd971237" candidate_uuid: "d61d9a305e944746a1a2d267e043628c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dcd1e7651844b8bc3042932d347c92" is_pre_election: true
I20260620 00:12:23.877600 4616 raft_consensus.cc:2454] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d61d9a305e944746a1a2d267e043628c in term 0.
I20260620 00:12:23.877629 4683 leader_election.cc:302] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4c05c5b60540649aa5935f185d7675, d61d9a305e944746a1a2d267e043628c; no voters:
I20260620 00:12:23.877805 4952 raft_consensus.cc:2781] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:23.877862 4952 raft_consensus.cc:482] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:23.877882 4952 raft_consensus.cc:3037] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:23.878614 4952 raft_consensus.cc:504] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.878772 4952 leader_election.cc:288] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [CANDIDATE]: Term 1 election: Requested vote from peers ba4c05c5b60540649aa5935f185d7675 (127.0.98.3:42803), 08dcd1e7651844b8bc3042932d347c92 (127.0.98.1:34159)
I20260620 00:12:23.878978 4616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44abfbc1f056401b8e57c651bd971237" candidate_uuid: "d61d9a305e944746a1a2d267e043628c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dcd1e7651844b8bc3042932d347c92"
I20260620 00:12:23.878978 4882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44abfbc1f056401b8e57c651bd971237" candidate_uuid: "d61d9a305e944746a1a2d267e043628c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba4c05c5b60540649aa5935f185d7675"
I20260620 00:12:23.879048 4616 raft_consensus.cc:3037] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:23.879082 4882 raft_consensus.cc:3037] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:23.879860 4616 raft_consensus.cc:2454] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61d9a305e944746a1a2d267e043628c in term 1.
I20260620 00:12:23.879870 4882 raft_consensus.cc:2454] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d61d9a305e944746a1a2d267e043628c in term 1.
I20260620 00:12:23.880053 4683 leader_election.cc:302] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4c05c5b60540649aa5935f185d7675, d61d9a305e944746a1a2d267e043628c; no voters:
I20260620 00:12:23.880215 4952 raft_consensus.cc:2781] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:23.880414 4952 raft_consensus.cc:686] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 1 LEADER]: Becoming Leader. State: Replica: d61d9a305e944746a1a2d267e043628c, State: Running, Role: LEADER
I20260620 00:12:23.880525 4952 consensus_queue.cc:237] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } }
I20260620 00:12:23.881449 4472 catalog_manager.cc:5495] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c reported cstate change: term changed from 0 to 1, leader changed from <none> to d61d9a305e944746a1a2d267e043628c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d61d9a305e944746a1a2d267e043628c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d61d9a305e944746a1a2d267e043628c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:23.911337 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:23.908815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.98.2:35427)
0620 00:12:23.908888 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:23.908890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:23.908900 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:23.908947 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:23.908947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:23.908966 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:23.909045 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:23.909173 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.909656 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.910484 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:23.910684 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:23.910762 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:23.910951 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:23.911066 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:23.911161 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:23.911216 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 00:12:23.911237 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:12:23.913502 4616 raft_consensus.cc:1261] T 44abfbc1f056401b8e57c651bd971237 P 08dcd1e7651844b8bc3042932d347c92 [term 1 FOLLOWER]: Refusing update from remote peer d61d9a305e944746a1a2d267e043628c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:23.913502 4882 raft_consensus.cc:1261] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 1 FOLLOWER]: Refusing update from remote peer d61d9a305e944746a1a2d267e043628c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:23.913895 4955 consensus_queue.cc:1035] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08dcd1e7651844b8bc3042932d347c92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:23.914033 4952 consensus_queue.cc:1035] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba4c05c5b60540649aa5935f185d7675" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:23.918658 4471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:23.918785 4471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:23.919579 4471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5a66b346db847559db7f9f9b5a34dbb: Sending DeleteTablet for 3 replicas of tablet 44abfbc1f056401b8e57c651bd971237
I20260620 00:12:23.919943 4862 tablet_service.cc:1514] Processing DeleteTablet for tablet 44abfbc1f056401b8e57c651bd971237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52698
I20260620 00:12:23.919986 4729 tablet_service.cc:1514] Processing DeleteTablet for tablet 44abfbc1f056401b8e57c651bd971237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33578
I20260620 00:12:23.920193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4534
I20260620 00:12:23.920202 4596 tablet_service.cc:1514] Processing DeleteTablet for tablet 44abfbc1f056401b8e57c651bd971237 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38666
I20260620 00:12:23.920850 4968 tablet_replica.cc:330] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: stopping tablet replica
I20260620 00:12:23.920990 4968 raft_consensus.cc:2229] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:23.920997 4967 tablet_replica.cc:330] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: stopping tablet replica
I20260620 00:12:23.921082 4967 raft_consensus.cc:2229] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:23.921201 4967 raft_consensus.cc:2258] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:23.921197 4968 raft_consensus.cc:2258] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:23.921725 4967 ts_tablet_manager.cc:1905] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:23.921736 4968 ts_tablet_manager.cc:1905] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:23.923695 4967 ts_tablet_manager.cc:1918] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:23.923774 4967 log.cc:1196] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/44abfbc1f056401b8e57c651bd971237
I20260620 00:12:23.924032 4967 ts_tablet_manager.cc:1939] T 44abfbc1f056401b8e57c651bd971237 P ba4c05c5b60540649aa5935f185d7675: Deleting consensus metadata
I20260620 00:12:23.924160 4968 ts_tablet_manager.cc:1918] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:23.924239 4968 log.cc:1196] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/44abfbc1f056401b8e57c651bd971237
I20260620 00:12:23.924507 4968 ts_tablet_manager.cc:1939] T 44abfbc1f056401b8e57c651bd971237 P d61d9a305e944746a1a2d267e043628c: Deleting consensus metadata
I20260620 00:12:23.924571 4457 catalog_manager.cc:4841] TS ba4c05c5b60540649aa5935f185d7675 (127.0.98.3:42803): tablet 44abfbc1f056401b8e57c651bd971237 (table test-table [id=f9d0df9eb4d145cf8b79b990d1e05456]) successfully deleted
W20260620 00:12:23.928571 4458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34159 (error 108)
I20260620 00:12:23.929118 4458 catalog_manager.cc:4841] TS d61d9a305e944746a1a2d267e043628c (127.0.98.2:35427): tablet 44abfbc1f056401b8e57c651bd971237 (table test-table [id=f9d0df9eb4d145cf8b79b990d1e05456]) successfully deleted
W20260620 00:12:23.929204 4942 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:23.929060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34159 (local address 127.0.0.1:38678)
0620 00:12:23.929101 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:23.929143 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34159: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 00:12:23.929361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4667
W20260620 00:12:23.929385 4458 catalog_manager.cc:4584] TS 08dcd1e7651844b8bc3042932d347c92 (127.0.98.1:34159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44abfbc1f056401b8e57c651bd971237: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34159: connect: Connection reset by peer (error 104)
I20260620 00:12:23.934664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4800
I20260620 00:12:23.940191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4441
2026-06-20T00:12:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2094 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T00:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:23Z Disabled control of system clock
I20260620 00:12:23.969074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:12:24.059904 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:24.060190 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:24.060216 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:24.061530 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:24.061582 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:24.061597 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:24.061610 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:24.061622 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:24.063098 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:24.063313 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:24.063562 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:24.068454 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.068454 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.068485 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.068809 4973 server_base.cc:1034] running on GCE node
I20260620 00:12:24.069079 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:24.069408 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:24.070595 4973 hybrid_clock.cc:648] HybridClock initialized: now 1781914344070578 us; error 34 us; skew 500 ppm
I20260620 00:12:24.072053 4973 webserver.cc:460] Webserver started at http://127.0.98.62:46733/ using document root <none> and password file <none>
I20260620 00:12:24.072309 4973 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:24.072413 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:24.072597 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:24.073812 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "b277edce20f840449beda90a82e42e69"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "ff70dc52909f4055888ba93a816c7c21"
server_key_iv: "1ffd3742d36ea07e67568067c8645b7d"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.074188 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "b277edce20f840449beda90a82e42e69"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "ff70dc52909f4055888ba93a816c7c21"
server_key_iv: "1ffd3742d36ea07e67568067c8645b7d"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.075614 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:24.076478 4987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.076753 4973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:24.076861 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "b277edce20f840449beda90a82e42e69"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "ff70dc52909f4055888ba93a816c7c21"
server_key_iv: "1ffd3742d36ea07e67568067c8645b7d"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.076951 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:24.080711 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:24.081028 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:24.081179 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:24.085520 4973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33453
I20260620 00:12:24.085554 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33453 every 8 connection(s)
I20260620 00:12:24.085949 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:24.086663 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:24.089614 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: Bootstrap starting.
I20260620 00:12:24.090266 5040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:24.090570 5040 log.cc:824] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:24.091370 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: No bootstrap required, opened a new log
I20260620 00:12:24.093036 5040 raft_consensus.cc:348] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260620 00:12:24.093189 5040 raft_consensus.cc:374] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:24.093238 5040 raft_consensus.cc:729] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b277edce20f840449beda90a82e42e69, State: Initialized, Role: FOLLOWER
I20260620 00:12:24.093405 5040 consensus_queue.cc:260] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260620 00:12:24.093505 5040 raft_consensus.cc:388] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:24.093528 5040 raft_consensus.cc:482] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:24.093576 5040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:24.094092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 4973
I20260620 00:12:24.094199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:24.094377 392 external_mini_cluster.cc:1428] Setting key d55af678bab56a7fa2a18310ab46560b
I20260620 00:12:24.096582 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.094910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38350 (local address 127.0.98.62:33453)
0620 00:12:24.095150 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.095158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.095184 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:12:24.095289 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.095295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.095331 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.095438 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.096108 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.096112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.096112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.096116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.096153 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.096164 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.096230 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.096343 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.096391 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":143,"threads_started":1}
I20260620 00:12:24.134801 5040 raft_consensus.cc:504] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260620 00:12:24.135015 5040 leader_election.cc:302] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b277edce20f840449beda90a82e42e69; no voters:
I20260620 00:12:24.135262 5040 leader_election.cc:288] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:24.135329 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:24.135497 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [term 1 LEADER]: Becoming Leader. State: Replica: b277edce20f840449beda90a82e42e69, State: Running, Role: LEADER
I20260620 00:12:24.135567 5040 sys_catalog.cc:564] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:24.135605 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260620 00:12:24.135927 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b277edce20f840449beda90a82e42e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260620 00:12:24.135991 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:24.136099 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b277edce20f840449beda90a82e42e69. Latest consensus state: current_term: 1 leader_uuid: "b277edce20f840449beda90a82e42e69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b277edce20f840449beda90a82e42e69" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260620 00:12:24.136161 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:24.136374 5050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:24.136904 5050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:24.138877 5050 catalog_manager.cc:1347] Generated new cluster ID: 66787e161cb94b7c97d1c5627b7e7102
I20260620 00:12:24.138942 5050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:24.147900 5050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:24.148103 5050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:24.151136 5050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: Generated new TSK 0
I20260620 00:12:24.151348 5050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:12:24.160393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--builtin_ntp_servers=127.0.98.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:12:24.253578 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:24.253834 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:24.253875 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:24.255118 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:24.255190 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:24.255236 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:24.256764 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:24.257038 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:24.257403 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:24.263659 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.263780 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.263661 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.264381 5064 server_base.cc:1034] running on GCE node
I20260620 00:12:24.264616 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:24.264907 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:24.266081 5064 hybrid_clock.cc:648] HybridClock initialized: now 1781914344266049 us; error 30 us; skew 500 ppm
I20260620 00:12:24.267498 5064 webserver.cc:460] Webserver started at http://127.0.98.1:46243/ using document root <none> and password file <none>
I20260620 00:12:24.267769 5064 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:24.267840 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:24.268006 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:24.269213 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c8a8e577fd84cc5b69e754af9416d16"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "1359bec39eb83c8f8053fd37b4dd4a03"
server_key_iv: "d595d2108f041bfc28b29674211253e6"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.269583 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c8a8e577fd84cc5b69e754af9416d16"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "1359bec39eb83c8f8053fd37b4dd4a03"
server_key_iv: "d595d2108f041bfc28b29674211253e6"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.270980 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:24.271871 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.272044 5064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:24.272176 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "2c8a8e577fd84cc5b69e754af9416d16"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "1359bec39eb83c8f8053fd37b4dd4a03"
server_key_iv: "d595d2108f041bfc28b29674211253e6"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.272264 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:24.275854 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:24.276153 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:24.276306 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:24.276515 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:24.276844 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:24.276877 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.276897 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:24.276911 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.283085 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34473
I20260620 00:12:24.283123 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34473 every 8 connection(s)
I20260620 00:12:24.283479 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:24.286067 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5064
I20260620 00:12:24.286159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:24.286350 392 external_mini_cluster.cc:1428] Setting key 397394e9b49216a5aa79d71d9ef76029
I20260620 00:12:24.286880 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.284338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49915 (local address 127.0.98.62:33453)
0620 00:12:24.284405 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.284408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.284674 (+ 266us) server_negotiation.cc:407] Connection header received
0620 00:12:24.285210 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.285211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.285220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.285286 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.286602 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.286604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.286604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.286608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.286625 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.286631 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.286717 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.286790 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.286806 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:12:24.287074 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.284309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.1:49915)
0620 00:12:24.284505 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.284573 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.285088 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.285357 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.285361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.285371 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.286416 (+ 1045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.286417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.286454 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.286492 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.286718 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.286724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.286726 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.286797 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.286846 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260620 00:12:24.287439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--builtin_ntp_servers=127.0.98.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 00:12:24.287647 5192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260620 00:12:24.287739 5192 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:24.287894 5192 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260620 00:12:24.288394 5004 ts_manager.cc:194] Registered new tserver with Master: 2c8a8e577fd84cc5b69e754af9416d16 (127.0.98.1:34473)
W20260620 00:12:24.371549 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:24.371773 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:24.371795 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:24.373208 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:24.373270 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:24.373306 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:24.374871 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:24.375178 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:24.375502 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:24.380685 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.380685 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.380844 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.380750 5195 server_base.cc:1034] running on GCE node
I20260620 00:12:24.381111 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:24.381381 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:24.382550 5195 hybrid_clock.cc:648] HybridClock initialized: now 1781914344382525 us; error 41 us; skew 500 ppm
I20260620 00:12:24.383935 5195 webserver.cc:460] Webserver started at http://127.0.98.2:36155/ using document root <none> and password file <none>
I20260620 00:12:24.384196 5195 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:24.384271 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:24.384433 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:24.385587 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "136cfd7609be4819add3749e833911fc"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "c78acac3dc192617e7c622a189c11ec1"
server_key_iv: "a1bbbb79a7709abdc8b153344784d211"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.385967 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "136cfd7609be4819add3749e833911fc"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "c78acac3dc192617e7c622a189c11ec1"
server_key_iv: "a1bbbb79a7709abdc8b153344784d211"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.387310 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:24.388074 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.388262 5195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:24.388365 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "136cfd7609be4819add3749e833911fc"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "c78acac3dc192617e7c622a189c11ec1"
server_key_iv: "a1bbbb79a7709abdc8b153344784d211"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.388440 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:24.391837 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:24.392110 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:24.392254 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:24.392458 5195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:24.392804 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:24.392859 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.392899 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:24.392925 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.398636 5195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33723
I20260620 00:12:24.398686 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33723 every 8 connection(s)
I20260620 00:12:24.399021 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:24.401918 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.399763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59839 (local address 127.0.98.62:33453)
0620 00:12:24.399810 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.399812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.400032 (+ 220us) server_negotiation.cc:407] Connection header received
0620 00:12:24.400466 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.400466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.400471 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.400593 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.401707 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.401709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.401709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.401713 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.401729 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.401733 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.401791 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.401850 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.401863 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:12:24.402083 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.399720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.2:59839)
0620 00:12:24.399897 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.399938 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.400388 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.400677 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.400681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.400688 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.401563 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.401564 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.401592 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.401620 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.401789 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.401792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.401794 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.401844 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.401890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 00:12:24.402547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5195
I20260620 00:12:24.402611 5323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260620 00:12:24.402661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:24.402714 5323 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:24.402868 392 external_mini_cluster.cc:1428] Setting key eda0e0e9f6330c3dcdec088ba3eb34eb
I20260620 00:12:24.402889 5323 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260620 00:12:24.403328 5003 ts_manager.cc:194] Registered new tserver with Master: 136cfd7609be4819add3749e833911fc (127.0.98.2:33723)
I20260620 00:12:24.403936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--builtin_ntp_servers=127.0.98.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:12:24.489156 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:24.489375 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:24.489399 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:24.490597 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:24.490638 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:24.490664 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:24.492210 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:24.492427 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:24.492767 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:24.497994 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.497998 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.498080 5326 server_base.cc:1034] running on GCE node
W20260620 00:12:24.498034 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.498569 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:24.498842 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:24.499996 5326 hybrid_clock.cc:648] HybridClock initialized: now 1781914344499985 us; error 33 us; skew 500 ppm
I20260620 00:12:24.501443 5326 webserver.cc:460] Webserver started at http://127.0.98.3:37369/ using document root <none> and password file <none>
I20260620 00:12:24.501762 5326 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:24.501816 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:24.501955 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:24.503082 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "230d9d4171ca4590b194a569039af4bb"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "201b5f78a83421b88ba9b62ff69b9c16"
server_key_iv: "68c2aca228e404639430fbec4e71eeb5"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.503445 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "230d9d4171ca4590b194a569039af4bb"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "201b5f78a83421b88ba9b62ff69b9c16"
server_key_iv: "68c2aca228e404639430fbec4e71eeb5"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.504886 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:24.505744 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.505934 5326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:24.506075 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "230d9d4171ca4590b194a569039af4bb"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "201b5f78a83421b88ba9b62ff69b9c16"
server_key_iv: "68c2aca228e404639430fbec4e71eeb5"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.506148 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:24.509758 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:24.510035 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:24.510170 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:24.510366 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:24.510669 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:24.510722 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.510762 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:24.510789 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.516424 5326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41939
I20260620 00:12:24.516495 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41939 every 8 connection(s)
I20260620 00:12:24.516857 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:24.518718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5326
I20260620 00:12:24.518832 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:24.519016 392 external_mini_cluster.cc:1428] Setting key 0a317552821e0b92a1839c05dcb1b63c
I20260620 00:12:24.520153 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.517644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36341 (local address 127.0.98.62:33453)
0620 00:12:24.517694 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.517696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.517931 (+ 235us) server_negotiation.cc:407] Connection header received
0620 00:12:24.518464 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.518466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.518471 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.518532 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.519941 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.519942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.519942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.519947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.519961 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.519965 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.520022 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.520089 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.520101 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:24.520351 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.517646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.3:36341)
0620 00:12:24.517816 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.517848 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.518349 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.518624 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.518629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.518639 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.519759 (+ 1120us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.519759 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.519796 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.519829 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.520021 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.520026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.520029 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.520092 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.520144 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260620 00:12:24.520820 5454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260620 00:12:24.520925 5454 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:24.521067 5454 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260620 00:12:24.521406 5004 ts_manager.cc:194] Registered new tserver with Master: 230d9d4171ca4590b194a569039af4bb (127.0.98.3:41939)
I20260620 00:12:24.531008 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:12:24.533550 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.532450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.0.98.62:33453)
0620 00:12:24.532511 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.532513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.532628 (+ 115us) server_negotiation.cc:407] Connection header received
0620 00:12:24.532685 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.532685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.532690 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.532742 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.533339 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.533340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.533340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.533344 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.533359 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.533364 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.533409 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.533484 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.533497 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:12:24.535014 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:24.536720 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:24.540524 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.538817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34473 (local address 127.0.0.1:51446)
0620 00:12:24.539058 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.539068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.539103 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.539428 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.539429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.539432 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.540058 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.540059 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.540075 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.540100 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.540348 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.540354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.540357 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.540431 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.540446 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260620 00:12:24.540613 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.539051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.98.1:34473)
0620 00:12:24.539265 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.539270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.539281 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:24.539309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.539311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.539323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.539638 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.540190 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.540198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.540198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.540203 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.540241 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.540269 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.540330 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.540457 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.540475 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":107,"threads_started":1}
I20260620 00:12:24.540782 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.538802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41939 (local address 127.0.0.1:43150)
0620 00:12:24.538919 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.538961 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.539029 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.539402 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.539408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.539415 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.540145 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.540146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.540154 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.540167 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.540655 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.540656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.540657 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.540707 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.540721 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260620 00:12:24.541020 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.538982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.98.3:41939)
0620 00:12:24.539277 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.539282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.539294 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:24.539321 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.539323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.539332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.539402 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.540463 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.540467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.540467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.540471 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.540501 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.540514 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.540636 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.540907 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.540928 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":93,"threads_started":1}
I20260620 00:12:24.541452 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 060dd0d2b5574b1284705649babb0c68 (DEFAULT_TABLE table=test-table [id=dcbe97a0e9b444368cd4a0f0e1c1ea28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:24.541579 5388 tablet_service.cc:1467] Processing CreateTablet for tablet 060dd0d2b5574b1284705649babb0c68 (DEFAULT_TABLE table=test-table [id=dcbe97a0e9b444368cd4a0f0e1c1ea28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:24.541857 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060dd0d2b5574b1284705649babb0c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:24.541904 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060dd0d2b5574b1284705649babb0c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:24.544355 5468 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Bootstrap starting.
I20260620 00:12:24.544587 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.538843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33723 (local address 127.0.0.1:56982)
0620 00:12:24.539258 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.539269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.539320 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.539840 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.539841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.539843 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.540370 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.540370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.540375 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.540385 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.544412 (+ 4027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.544413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.544414 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.544458 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.544472 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":74,"threads_started":1}
I20260620 00:12:24.544813 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.539260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.0.98.2:33723)
0620 00:12:24.539456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.539460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.539469 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:24.539679 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.539688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.539697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.539767 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.540463 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.540469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.540469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.540473 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.540515 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.540565 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.544674 (+ 4109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.544706 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.544724 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260620 00:12:24.545018 5469 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: Bootstrap starting.
I20260620 00:12:24.545101 5468 tablet_bootstrap.cc:654] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:24.545434 5468 log.cc:824] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:24.545548 5469 tablet_bootstrap.cc:654] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:24.545526 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 060dd0d2b5574b1284705649babb0c68 (DEFAULT_TABLE table=test-table [id=dcbe97a0e9b444368cd4a0f0e1c1ea28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:24.545825 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060dd0d2b5574b1284705649babb0c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:24.545857 5469 log.cc:824] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:24.546245 5468 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: No bootstrap required, opened a new log
I20260620 00:12:24.546303 5468 ts_tablet_manager.cc:1397] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:12:24.546520 5469 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: No bootstrap required, opened a new log
I20260620 00:12:24.546561 5469 ts_tablet_manager.cc:1397] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:24.548472 5469 raft_consensus.cc:348] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.548682 5469 raft_consensus.cc:374] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:24.548718 5469 raft_consensus.cc:729] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8a8e577fd84cc5b69e754af9416d16, State: Initialized, Role: FOLLOWER
I20260620 00:12:24.548848 5469 consensus_queue.cc:260] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.549131 5469 ts_tablet_manager.cc:1428] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:24.549263 5192 heartbeater.cc:503] Master 127.0.98.62:33453 was elected leader, sending a full tablet report...
I20260620 00:12:24.549064 5468 raft_consensus.cc:348] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.549528 5468 raft_consensus.cc:374] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:24.549582 5468 raft_consensus.cc:729] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230d9d4171ca4590b194a569039af4bb, State: Initialized, Role: FOLLOWER
I20260620 00:12:24.549706 5468 consensus_queue.cc:260] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.550004 5468 ts_tablet_manager.cc:1428] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 00:12:24.550068 5454 heartbeater.cc:503] Master 127.0.98.62:33453 was elected leader, sending a full tablet report...
I20260620 00:12:24.550187 5472 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Bootstrap starting.
I20260620 00:12:24.550884 5472 tablet_bootstrap.cc:654] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:24.551198 5472 log.cc:824] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:24.551995 5472 tablet_bootstrap.cc:492] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: No bootstrap required, opened a new log
I20260620 00:12:24.552076 5472 ts_tablet_manager.cc:1397] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:12:24.553922 5472 raft_consensus.cc:348] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.554076 5472 raft_consensus.cc:374] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:24.554126 5472 raft_consensus.cc:729] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136cfd7609be4819add3749e833911fc, State: Initialized, Role: FOLLOWER
I20260620 00:12:24.554240 5472 consensus_queue.cc:260] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.554514 5472 ts_tablet_manager.cc:1428] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:12:24.554591 5323 heartbeater.cc:503] Master 127.0.98.62:33453 was elected leader, sending a full tablet report...
I20260620 00:12:24.637733 5476 raft_consensus.cc:482] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:24.637861 5476 raft_consensus.cc:504] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.638278 5476 leader_election.cc:288] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 230d9d4171ca4590b194a569039af4bb (127.0.98.3:41939), 2c8a8e577fd84cc5b69e754af9416d16 (127.0.98.1:34473)
I20260620 00:12:24.639736 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.638445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41939 (local address 127.0.98.2:34041)
0620 00:12:24.638513 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.638527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.638570 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.638809 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.638810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.638813 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.639329 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.639330 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.639338 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.639352 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.639555 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.639556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.639557 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.639640 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.639655 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 00:12:24.639778 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.638600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34041 (local address 127.0.98.3:41939)
0620 00:12:24.638665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.638667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.638679 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:24.638715 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.638716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.638721 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.638802 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.639452 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.639454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.639454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.639460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.639482 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.639488 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.639578 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.639658 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.639671 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:12:24.639860 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.638510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34473 (local address 127.0.98.2:41113)
0620 00:12:24.638705 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.638717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.638764 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.638919 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.638920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.638923 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:24.639509 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:12:24.639510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:24.639516 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:12:24.639528 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:24.639710 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:24.639711 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:24.639712 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:12:24.639758 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 00:12:24.639770 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260620 00:12:24.639860 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.638652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41113 (local address 127.0.98.1:34473)
0620 00:12:24.638734 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.638736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.638748 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:12:24.638835 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.638836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.638840 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.638915 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.639617 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.639618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.639619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.639624 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.639640 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.639646 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.639705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.639767 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.639780 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 00:12:24.640110 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060dd0d2b5574b1284705649babb0c68" candidate_uuid: "136cfd7609be4819add3749e833911fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230d9d4171ca4590b194a569039af4bb" is_pre_election: true
I20260620 00:12:24.640174 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060dd0d2b5574b1284705649babb0c68" candidate_uuid: "136cfd7609be4819add3749e833911fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8a8e577fd84cc5b69e754af9416d16" is_pre_election: true
I20260620 00:12:24.640268 5408 raft_consensus.cc:2454] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136cfd7609be4819add3749e833911fc in term 0.
I20260620 00:12:24.640301 5146 raft_consensus.cc:2454] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136cfd7609be4819add3749e833911fc in term 0.
I20260620 00:12:24.640575 5211 leader_election.cc:302] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136cfd7609be4819add3749e833911fc, 230d9d4171ca4590b194a569039af4bb; no voters:
I20260620 00:12:24.640722 5476 raft_consensus.cc:2781] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:24.640774 5476 raft_consensus.cc:482] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:24.640796 5476 raft_consensus.cc:3037] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:24.641494 5476 raft_consensus.cc:504] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.641640 5476 leader_election.cc:288] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [CANDIDATE]: Term 1 election: Requested vote from peers 230d9d4171ca4590b194a569039af4bb (127.0.98.3:41939), 2c8a8e577fd84cc5b69e754af9416d16 (127.0.98.1:34473)
I20260620 00:12:24.641767 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060dd0d2b5574b1284705649babb0c68" candidate_uuid: "136cfd7609be4819add3749e833911fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "230d9d4171ca4590b194a569039af4bb"
I20260620 00:12:24.641783 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060dd0d2b5574b1284705649babb0c68" candidate_uuid: "136cfd7609be4819add3749e833911fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c8a8e577fd84cc5b69e754af9416d16"
I20260620 00:12:24.641844 5408 raft_consensus.cc:3037] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:24.641848 5146 raft_consensus.cc:3037] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:24.642709 5408 raft_consensus.cc:2454] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136cfd7609be4819add3749e833911fc in term 1.
I20260620 00:12:24.642709 5146 raft_consensus.cc:2454] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136cfd7609be4819add3749e833911fc in term 1.
I20260620 00:12:24.642917 5212 leader_election.cc:302] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136cfd7609be4819add3749e833911fc, 2c8a8e577fd84cc5b69e754af9416d16; no voters:
I20260620 00:12:24.643015 5476 raft_consensus.cc:2781] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:24.643185 5476 raft_consensus.cc:686] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 1 LEADER]: Becoming Leader. State: Replica: 136cfd7609be4819add3749e833911fc, State: Running, Role: LEADER
I20260620 00:12:24.643291 5476 consensus_queue.cc:237] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } }
I20260620 00:12:24.644121 5003 catalog_manager.cc:5495] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 136cfd7609be4819add3749e833911fc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "136cfd7609be4819add3749e833911fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136cfd7609be4819add3749e833911fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33723 } health_report { overall_health: HEALTHY } } }
W20260620 00:12:24.649576 5324 tablet.cc:2369] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:12:24.657657 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.656612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.98.2:33723)
0620 00:12:24.656662 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.656664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.656673 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:24.656710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.656711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.656725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.656780 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.657395 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.657396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.657397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.657402 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.657419 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.657423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.657502 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.657564 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.657578 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:12:24.659516 5146 raft_consensus.cc:1261] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 1 FOLLOWER]: Refusing update from remote peer 136cfd7609be4819add3749e833911fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:24.659516 5408 raft_consensus.cc:1261] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 1 FOLLOWER]: Refusing update from remote peer 136cfd7609be4819add3749e833911fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:24.659904 5478 consensus_queue.cc:1035] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c8a8e577fd84cc5b69e754af9416d16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:24.660043 5476 consensus_queue.cc:1035] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "230d9d4171ca4590b194a569039af4bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:24.664032 5483 mvcc.cc:205] Tried to move back new op lower bound from 7298721155722756096 to 7298721155659706368. Current Snapshot: MvccSnapshot[applied={T|T < 7298721155722756096 or (T in {7298721155722756096})}]
I20260620 00:12:24.665446 5004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:24.665549 5004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:24.666326 5004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b277edce20f840449beda90a82e42e69: Sending DeleteTablet for 3 replicas of tablet 060dd0d2b5574b1284705649babb0c68
I20260620 00:12:24.666658 5388 tablet_service.cc:1514] Processing DeleteTablet for tablet 060dd0d2b5574b1284705649babb0c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:24 UTC) from {username='slave'} at 127.0.0.1:43150
I20260620 00:12:24.666679 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 060dd0d2b5574b1284705649babb0c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:24 UTC) from {username='slave'} at 127.0.0.1:51446
I20260620 00:12:24.666894 5491 tablet_replica.cc:330] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16: stopping tablet replica
I20260620 00:12:24.666931 5257 tablet_service.cc:1514] Processing DeleteTablet for tablet 060dd0d2b5574b1284705649babb0c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:24 UTC) from {username='slave'} at 127.0.0.1:56982
I20260620 00:12:24.666996 5491 raft_consensus.cc:2229] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:24.667104 5491 raft_consensus.cc:2258] T 060dd0d2b5574b1284705649babb0c68 P 2c8a8e577fd84cc5b69e754af9416d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:24.667279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5064
I20260620 00:12:24.668109 5493 tablet_replica.cc:330] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc: stopping tablet replica
I20260620 00:12:24.668267 5492 tablet_replica.cc:330] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: stopping tablet replica
I20260620 00:12:24.668434 5492 raft_consensus.cc:2229] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:24.668609 5493 raft_consensus.cc:2229] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:24.668779 5492 raft_consensus.cc:2258] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:24.668927 5493 raft_consensus.cc:2258] T 060dd0d2b5574b1284705649babb0c68 P 136cfd7609be4819add3749e833911fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:24.670167 5492 ts_tablet_manager.cc:1905] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:24.672091 5492 ts_tablet_manager.cc:1918] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:24.672179 5492 log.cc:1196] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/060dd0d2b5574b1284705649babb0c68
I20260620 00:12:24.672453 5492 ts_tablet_manager.cc:1939] T 060dd0d2b5574b1284705649babb0c68 P 230d9d4171ca4590b194a569039af4bb: Deleting consensus metadata
I20260620 00:12:24.673038 4989 catalog_manager.cc:4841] TS 230d9d4171ca4590b194a569039af4bb (127.0.98.3:41939): tablet 060dd0d2b5574b1284705649babb0c68 (table test-table [id=dcbe97a0e9b444368cd4a0f0e1c1ea28]) successfully deleted
I20260620 00:12:24.673151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5195
W20260620 00:12:24.673046 4990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34473 (error 108)
I20260620 00:12:24.674090 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.673954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34473 (local address 127.0.0.1:51462)
0620 00:12:24.674007 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.674047 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:12:24.674216 4990 catalog_manager.cc:4584] TS 2c8a8e577fd84cc5b69e754af9416d16 (127.0.98.1:34473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 060dd0d2b5574b1284705649babb0c68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34473: connect: Connection refused (error 111)
I20260620 00:12:24.678320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5326
I20260620 00:12:24.680388 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.678410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33723 (local address 127.0.0.1:57004)
0620 00:12:24.680321 (+ 1911us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.680353 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1888}
I20260620 00:12:24.683586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 4973
2026-06-20T00:12:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (742 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 00:12:24.703778 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:12:24.704933 392 webserver.cc:460] Webserver started at https://127.0.0.1:33439/ using document root <none> and password file <none>
I20260620 00:12:24.705052 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:12:24.705255 392 webserver.cc:460] Webserver started at http://127.0.0.1:45279/ using document root <none> and password file <none>
2026-06-20T00:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:24Z Disabled control of system clock
I20260620 00:12:24.713829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44947
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44947
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/testchainca.cert with env {}
W20260620 00:12:24.794545 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:24.794772 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:24.794795 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:24.795953 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:24.796000 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:24.796013 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:24.796026 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:24.796039 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:12:24.796052 5500 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:12:24.796063 5500 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33439/jwks.json
I20260620 00:12:24.797632 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:24.797919 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:24.798106 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:24.803418 5500 server_base.cc:1034] running on GCE node
W20260620 00:12:24.803419 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.803418 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:24.803601 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:24.803870 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:24.804265 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:24.805483 5500 hybrid_clock.cc:648] HybridClock initialized: now 1781914344805451 us; error 49 us; skew 500 ppm
I20260620 00:12:24.806770 5500 webserver.cc:460] Webserver started at http://127.0.98.62:43013/ using document root <none> and password file <none>
I20260620 00:12:24.807042 5500 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:24.807157 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:24.807307 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:24.808487 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "b3ab8d7c40c94b788560331125573a96"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "fb6cec8c802b28f51fbb8cb2d015acb2"
server_key_iv: "c3a1066d468668a543146e5e6fbb4a1b"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.808939 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "b3ab8d7c40c94b788560331125573a96"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "fb6cec8c802b28f51fbb8cb2d015acb2"
server_key_iv: "c3a1066d468668a543146e5e6fbb4a1b"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.810490 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:24.811444 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:24.811794 5500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:24.811969 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "b3ab8d7c40c94b788560331125573a96"
format_stamp: "Formatted at 2026-06-20 00:12:24 on dist-test-slave-28bv"
server_key: "fb6cec8c802b28f51fbb8cb2d015acb2"
server_key_iv: "c3a1066d468668a543146e5e6fbb4a1b"
server_key_version: "encryptionkey@0"
I20260620 00:12:24.812114 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:24.820858 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:24.821220 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:24.821376 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:24.825981 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44947
I20260620 00:12:24.826016 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44947 every 8 connection(s)
I20260620 00:12:24.826396 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:24.827445 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:24.828383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5500
I20260620 00:12:24.828488 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:24.828734 392 external_mini_cluster.cc:1428] Setting key d146c6a6aa0102df3591a698fa3f8698
I20260620 00:12:24.830958 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96: Bootstrap starting.
I20260620 00:12:24.831691 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:24.832006 5567 log.cc:824] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:24.832850 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96: No bootstrap required, opened a new log
I20260620 00:12:24.834671 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.829306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.0.98.62:44947)
0620 00:12:24.829509 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.829517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.829544 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:12:24.829967 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.829974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.830039 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.830185 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:24.830484 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.831594 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.832664 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.833143 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.833249 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:24.833960 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:24.833964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:24.833964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:24.834133 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:24.834167 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:24.834178 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:24.834231 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:24.834332 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 00:12:24.834403 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 00:12:24.835283 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } }
I20260620 00:12:24.835479 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:24.835508 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ab8d7c40c94b788560331125573a96, State: Initialized, Role: FOLLOWER
I20260620 00:12:24.835670 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } }
I20260620 00:12:24.835783 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:24.835827 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:24.835798 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:12:24.835886 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:24.836758 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } }
I20260620 00:12:24.836889 5567 leader_election.cc:302] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ab8d7c40c94b788560331125573a96; no voters:
I20260620 00:12:24.837081 5567 leader_election.cc:288] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:24.837138 5572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:24.837350 5572 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [term 1 LEADER]: Becoming Leader. State: Replica: b3ab8d7c40c94b788560331125573a96, State: Running, Role: LEADER
I20260620 00:12:24.837368 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:24.837471 5572 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } }
I20260620 00:12:24.837857 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ab8d7c40c94b788560331125573a96. Latest consensus state: current_term: 1 leader_uuid: "b3ab8d7c40c94b788560331125573a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } } }
I20260620 00:12:24.837936 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:24.837839 5573 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ab8d7c40c94b788560331125573a96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ab8d7c40c94b788560331125573a96" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44947 } } }
I20260620 00:12:24.838020 5573 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:24.838297 5579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:24.839027 5579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:24.840641 5579 catalog_manager.cc:1347] Generated new cluster ID: fbd1352648bb4555a2cb2b8d52c10268
I20260620 00:12:24.840708 5579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:24.858531 5579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:24.859278 5579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:24.869268 5579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ab8d7c40c94b788560331125573a96: Generated new TSK 0
I20260620 00:12:24.869515 5579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:12:24.900074 5597 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 00:12:24.900213 5597 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 00:12:24.900804 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:24.884229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44947 (local address 127.0.0.1:42278)
0620 00:12:24.884471 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:24.884482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:24.884559 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:24.884797 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:24.884798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:24.884800 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:12:24.884859 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:24.884861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.885828 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.885828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:24.886478 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:24.886481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.886994 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.886998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:24.887047 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:24.887067 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:12:24.900308 (+ 13241us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:12:24.900418 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44947: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
I20260620 00:12:24.901120 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:24.884328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.0.98.62:44947)
0620 00:12:24.884395 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:24.884397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:24.884561 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:12:24.884688 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:24.884690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:24.884702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:24.884784 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:12:24.884927 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.885724 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.886565 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:24.886881 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:24.886955 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:24.887180 (+ 225us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:12:24.900141 (+ 12961us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:12:24.901052 (+ 911us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 00:12:24.901170 5570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42278: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44947: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 00:12:24.901468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5500
2026-06-20T00:12:24Z chronyd exiting
I20260620 00:12:24.937589 392 test_util.cc:175] -----------------------------------------------
I20260620 00:12:24.937708 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914322045185-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 00:12:24.938644 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:12:24.939425 392 webserver.cc:460] Webserver started at https://127.0.0.1:45281/ using document root <none> and password file <none>
I20260620 00:12:24.939503 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:12:24.939764 392 webserver.cc:460] Webserver started at http://127.0.0.1:40687/ using document root <none> and password file <none>
2026-06-20T00:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:24Z Disabled control of system clock
I20260620 00:12:24.948844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41075
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41075
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45281/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/testchainca.cert with env {}
W20260620 00:12:25.033380 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:25.033631 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:25.033659 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:25.034971 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:25.035025 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:25.035038 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:25.035051 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:25.035064 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:12:25.035077 5603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:12:25.035126 5603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45281/jwks.json
I20260620 00:12:25.036823 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45281/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:25.037117 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:25.037367 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:25.043184 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:25.043179 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:25.043179 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:25.043248 5603 server_base.cc:1034] running on GCE node
I20260620 00:12:25.043787 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:25.044122 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:25.045308 5603 hybrid_clock.cc:648] HybridClock initialized: now 1781914345045300 us; error 56 us; skew 500 ppm
I20260620 00:12:25.046747 5603 webserver.cc:460] Webserver started at http://127.0.98.62:41221/ using document root <none> and password file <none>
I20260620 00:12:25.047029 5603 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:25.047109 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:25.047263 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:25.048388 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "d96f59973c5b4d198fada24cd7b61819"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dae449f581670b0aa909b93df9e2d526"
server_key_iv: "70f3722845c799109321fbb2b129258a"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.048822 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "d96f59973c5b4d198fada24cd7b61819"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dae449f581670b0aa909b93df9e2d526"
server_key_iv: "70f3722845c799109321fbb2b129258a"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.050256 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:25.051280 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:25.051529 5603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:25.051653 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "d96f59973c5b4d198fada24cd7b61819"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dae449f581670b0aa909b93df9e2d526"
server_key_iv: "70f3722845c799109321fbb2b129258a"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.051741 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:25.066390 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:25.066756 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:25.066928 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:25.071849 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41075
I20260620 00:12:25.071883 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41075 every 8 connection(s)
I20260620 00:12:25.072260 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:25.073271 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:25.073974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5603
I20260620 00:12:25.074081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:25.074273 392 external_mini_cluster.cc:1428] Setting key f0ce63dfab4d212083239317d3c8ff0c
I20260620 00:12:25.076196 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819: Bootstrap starting.
I20260620 00:12:25.076926 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:25.077298 5670 log.cc:824] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:25.078131 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819: No bootstrap required, opened a new log
I20260620 00:12:25.080237 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.074822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.0.98.62:41075)
0620 00:12:25.075062 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.075070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.075095 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:12:25.075542 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.075549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.075613 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.075767 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:25.075973 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.076993 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.078153 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.078628 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.078729 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.079474 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:25.079478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:25.079479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:25.079650 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:25.079683 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:25.079694 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:25.079753 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:25.079927 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 00:12:25.080054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260620 00:12:25.080504 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } }
I20260620 00:12:25.080731 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:25.080773 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96f59973c5b4d198fada24cd7b61819, State: Initialized, Role: FOLLOWER
I20260620 00:12:25.081025 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } }
I20260620 00:12:25.081139 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:25.081190 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:25.081241 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:25.081485 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:12:25.082062 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } }
I20260620 00:12:25.082190 5670 leader_election.cc:302] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96f59973c5b4d198fada24cd7b61819; no voters:
I20260620 00:12:25.082376 5670 leader_election.cc:288] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:25.082459 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:25.082623 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [term 1 LEADER]: Becoming Leader. State: Replica: d96f59973c5b4d198fada24cd7b61819, State: Running, Role: LEADER
I20260620 00:12:25.082639 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:25.082780 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } }
I20260620 00:12:25.083169 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d96f59973c5b4d198fada24cd7b61819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } } }
I20260620 00:12:25.083288 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.083169 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d96f59973c5b4d198fada24cd7b61819. Latest consensus state: current_term: 1 leader_uuid: "d96f59973c5b4d198fada24cd7b61819" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96f59973c5b4d198fada24cd7b61819" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41075 } } }
I20260620 00:12:25.083381 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.083719 5683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:25.084227 5683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:25.085768 5683 catalog_manager.cc:1347] Generated new cluster ID: 32116b4175ab4a36bdf5101a5538886f
I20260620 00:12:25.085826 5683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:25.097684 5683 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:25.098327 5683 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:25.109148 5683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d96f59973c5b4d198fada24cd7b61819: Generated new TSK 0
I20260620 00:12:25.109381 5683 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:12:25.138283 5700 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 00:12:25.138402 5700 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 00:12:25.138710 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.120474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.98.62:41075)
0620 00:12:25.120529 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.120531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.120648 (+ 117us) server_negotiation.cc:407] Connection header received
0620 00:12:25.120722 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.120723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.120736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.120828 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:12:25.120956 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.121786 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.122679 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.123012 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.123092 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.123314 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:12:25.138441 (+ 15127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45281/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:12:25.138630 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45281/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 00:12:25.138777 5673 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33586: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45281/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 00:12:25.138796 5699 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:25.120401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41075 (local address 127.0.0.1:33586)
0620 00:12:25.120558 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:25.120569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:25.120648 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:25.120837 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:25.120837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:25.120839 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:12:25.120891 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.120893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.121908 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.121909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.122580 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.122584 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.123122 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.123127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.123181 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.123187 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:12:25.138590 (+ 15403us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45281/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:12:25.138697 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45281/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260620 00:12:25.139072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5603
2026-06-20T00:12:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 00:12:25.178179 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:12:25.179044 392 webserver.cc:460] Webserver started at https://127.0.0.1:44117/ using document root <none> and password file <none>
I20260620 00:12:25.179109 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:12:25.179338 392 webserver.cc:460] Webserver started at http://127.0.0.1:38237/ using document root <none> and password file <none>
2026-06-20T00:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:25Z Disabled control of system clock
I20260620 00:12:25.187386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44117/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:12:25.269273 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:25.269520 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:25.269544 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:25.270836 5706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:25.270908 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:25.270921 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:25.270934 5706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:25.270947 5706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:12:25.271003 5706 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:12:25.271019 5706 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44117/jwks.json
I20260620 00:12:25.272469 5706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44117/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:25.272784 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:25.273105 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:25.278343 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:25.278364 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:25.278475 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:25.278630 5706 server_base.cc:1034] running on GCE node
I20260620 00:12:25.278919 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:25.279193 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:25.280346 5706 hybrid_clock.cc:648] HybridClock initialized: now 1781914345280323 us; error 38 us; skew 500 ppm
I20260620 00:12:25.282006 5706 webserver.cc:460] Webserver started at http://127.0.98.62:35535/ using document root <none> and password file <none>
I20260620 00:12:25.282222 5706 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:25.282265 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:25.282382 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:25.283522 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "22c6e0a67de89a2edd6345010c5cbdd4"
server_key_iv: "a3f1656841537d15119dc8b99d3fec25"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.283851 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "22c6e0a67de89a2edd6345010c5cbdd4"
server_key_iv: "a3f1656841537d15119dc8b99d3fec25"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.285297 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:25.286087 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:25.286284 5706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:25.286377 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "22c6e0a67de89a2edd6345010c5cbdd4"
server_key_iv: "a3f1656841537d15119dc8b99d3fec25"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.286434 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:25.303552 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:25.303881 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:25.304003 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:25.308225 5706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38683
I20260620 00:12:25.308288 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38683 every 8 connection(s)
I20260620 00:12:25.308703 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:25.309471 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:25.312263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5706
I20260620 00:12:25.312352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:25.312574 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: Bootstrap starting.
I20260620 00:12:25.312640 392 external_mini_cluster.cc:1428] Setting key 08ecca8c57c2b004f7496f2b267697fe
I20260620 00:12:25.313378 5773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:25.313719 5773 log.cc:824] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:25.314495 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: No bootstrap required, opened a new log
I20260620 00:12:25.316013 5773 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 00:12:25.316201 5773 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:25.316238 5773 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ad097a6ea14303a3f4ef3d806e0fbb, State: Initialized, Role: FOLLOWER
I20260620 00:12:25.316363 5773 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 00:12:25.316435 5773 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:25.316463 5773 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:25.316509 5773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:25.317471 5773 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 00:12:25.317602 5773 leader_election.cc:302] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7ad097a6ea14303a3f4ef3d806e0fbb; no voters:
I20260620 00:12:25.317797 5773 leader_election.cc:288] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:25.317827 5778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:25.317931 5778 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [term 1 LEADER]: Becoming Leader. State: Replica: c7ad097a6ea14303a3f4ef3d806e0fbb, State: Running, Role: LEADER
I20260620 00:12:25.318024 5778 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260620 00:12:25.318092 5773 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:25.318362 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260620 00:12:25.318444 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.318579 5778 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7ad097a6ea14303a3f4ef3d806e0fbb. Latest consensus state: current_term: 1 leader_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ad097a6ea14303a3f4ef3d806e0fbb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260620 00:12:25.318653 5778 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.319090 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.313416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.98.62:38683)
0620 00:12:25.313628 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.313635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.313661 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:12:25.314283 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.314292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.314355 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.314478 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:25.314729 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.315697 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.316841 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.317255 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.317356 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.318130 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:25.318134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:25.318135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:25.318375 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:25.318419 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:25.318439 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:25.318763 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:25.318798 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:12:25.318888 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
W20260620 00:12:25.320034 5794 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:25.320096 5794 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:25.320125 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:12:25.320269 5782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:25.320448 5782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:25.322053 5782 catalog_manager.cc:1347] Generated new cluster ID: 7f0a3a1afd6c4384be44cfe43566420b
I20260620 00:12:25.322115 5782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:25.336755 5782 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:25.337502 5782 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:25.343976 5782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7ad097a6ea14303a3f4ef3d806e0fbb: Generated new TSK 0
I20260620 00:12:25.344192 5782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:12:25.358366 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.354343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.98.62:38683)
0620 00:12:25.354421 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.354423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.354557 (+ 134us) server_negotiation.cc:407] Connection header received
0620 00:12:25.354623 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.354625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.354640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.354725 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:12:25.354859 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.355697 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.356517 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.356858 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.356933 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.357093 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:12:25.358127 (+ 1034us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:12:25.358267 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 00:12:25.358436 5776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35658: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 00:12:25.360312 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:25.354272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38683 (local address 127.0.0.1:35658)
0620 00:12:25.354474 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:25.354492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:25.354556 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:25.354740 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:25.354741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:25.354742 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:12:25.354799 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.354801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.355800 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.355801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.356426 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.356429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.356932 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.356940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.356981 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.356987 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:12:25.358229 (+ 1242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:12:25.360209 (+ 1980us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44117/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":1784,"thread_start_us":114,"threads_started":1}
I20260620 00:12:25.360705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5706
E20260620 00:12:25.371213 5803 webserver.cc:533] Webserver: sslize failed:
E20260620 00:12:25.371300 5803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T00:12:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T00:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:25Z Disabled control of system clock
I20260620 00:12:25.407202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37059
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37059
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/testchainca.cert with env {}
W20260620 00:12:25.493780 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:25.494009 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:25.494031 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:25.495306 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:25.495359 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:25.495373 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:25.495385 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:25.495397 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:12:25.495410 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:12:25.495453 5807 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 00:12:25.497150 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:25.497437 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:25.497682 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:25.502794 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:25.502827 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:25.503024 5807 server_base.cc:1034] running on GCE node
W20260620 00:12:25.502816 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:25.503346 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:25.503686 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:25.504859 5807 hybrid_clock.cc:648] HybridClock initialized: now 1781914345504813 us; error 69 us; skew 500 ppm
I20260620 00:12:25.506294 5807 webserver.cc:460] Webserver started at http://127.0.98.62:38601/ using document root <none> and password file <none>
I20260620 00:12:25.506561 5807 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:25.506637 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:25.506788 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:25.507879 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "ed7f55ae0d5b45f3a46af27b1492d873"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dcd3f6820d7fd1cea59d1a7ed32bc1e3"
server_key_iv: "01416c711761d85675985a4b0db1ab38"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.508262 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed7f55ae0d5b45f3a46af27b1492d873"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dcd3f6820d7fd1cea59d1a7ed32bc1e3"
server_key_iv: "01416c711761d85675985a4b0db1ab38"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.509706 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:25.510555 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:25.510737 5807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:25.510839 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "ed7f55ae0d5b45f3a46af27b1492d873"
format_stamp: "Formatted at 2026-06-20 00:12:25 on dist-test-slave-28bv"
server_key: "dcd3f6820d7fd1cea59d1a7ed32bc1e3"
server_key_iv: "01416c711761d85675985a4b0db1ab38"
server_key_version: "encryptionkey@0"
I20260620 00:12:25.510950 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:25.543764 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:25.544121 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:25.544283 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:25.548095 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37059
I20260620 00:12:25.548171 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37059 every 8 connection(s)
I20260620 00:12:25.548527 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:25.549531 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:25.551858 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873: Bootstrap starting.
I20260620 00:12:25.552425 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:25.552757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5807
I20260620 00:12:25.552815 5874 log.cc:824] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:25.552863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:25.553061 392 external_mini_cluster.cc:1428] Setting key f6f9dca82755fbe48fb73054f901ebc9
I20260620 00:12:25.553809 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873: No bootstrap required, opened a new log
I20260620 00:12:25.556450 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } }
I20260620 00:12:25.556658 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:25.556692 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7f55ae0d5b45f3a46af27b1492d873, State: Initialized, Role: FOLLOWER
I20260620 00:12:25.556836 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } }
I20260620 00:12:25.556926 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:25.556965 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:25.557016 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:25.557637 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } }
I20260620 00:12:25.557757 5874 leader_election.cc:302] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7f55ae0d5b45f3a46af27b1492d873; no voters:
I20260620 00:12:25.557943 5874 leader_election.cc:288] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:25.558010 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:25.558166 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [term 1 LEADER]: Becoming Leader. State: Replica: ed7f55ae0d5b45f3a46af27b1492d873, State: Running, Role: LEADER
I20260620 00:12:25.558233 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:25.558259 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } }
I20260620 00:12:25.558566 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7f55ae0d5b45f3a46af27b1492d873. Latest consensus state: current_term: 1 leader_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } } }
I20260620 00:12:25.558667 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.558911 5885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:25.559288 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.553651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.0.98.62:37059)
0620 00:12:25.553848 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.553855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.553880 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:12:25.554369 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.554376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.554434 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.554553 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:25.554839 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.555844 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.556987 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.557478 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.557585 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.558354 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:25.558368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:25.558369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:12:25.558579 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:25.558619 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:12:25.558633 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:25.558696 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:25.558992 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 00:12:25.559089 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260620 00:12:25.559652 5885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:25.559823 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7f55ae0d5b45f3a46af27b1492d873" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37059 } } }
I20260620 00:12:25.559908 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:25.560403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:12:25.561374 5885 catalog_manager.cc:1347] Generated new cluster ID: dc395b555c814ed1a1f031248800afaf
I20260620 00:12:25.561429 5885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:25.584632 5885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:25.585203 5885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:25.598310 5885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7f55ae0d5b45f3a46af27b1492d873: Generated new TSK 0
I20260620 00:12:25.598583 5885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:12:26.053977 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:25.612794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.0.98.62:37059)
0620 00:12:25.612865 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:25.612867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:25.613021 (+ 154us) server_negotiation.cc:407] Connection header received
0620 00:12:25.613126 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:25.613127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:25.613147 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:25.613230 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:12:25.613405 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.614000 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.614929 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.615145 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.615225 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.615425 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:12:26.053666 (+438241us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:12:26.053875 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:12:26.054065 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:25.612718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37059 (local address 127.0.0.1:58428)
0620 00:12:25.612873 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:25.612949 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:25.613042 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:25.613227 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:25.613228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:25.613230 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:12:25.613293 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.613295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.614102 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.614103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.614821 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:25.614824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:25.615229 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:25.615236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:25.615294 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:25.615299 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:12:26.053813 (+438514us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:12:26.053956 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260620 00:12:26.054072 5878 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 00:12:26.054414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5807
2026-06-20T00:12:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (672 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:26 dist-test-slave-28bv krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:26 dist-test-slave-28bv krb5kdc[5915](info): set up 2 sockets
Jun 20 00:12:26 dist-test-slave-28bv krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260620 00:12:27.344807 392 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:12:27.476269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:27.561272 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:27.561547 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:27.561590 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:27.562846 5928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:27.562928 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:27.562955 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:27.562984 5928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:27.563016 5928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:27.564718 5928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:27.565018 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:27.565297 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:27.571048 5928 server_base.cc:1034] running on GCE node
W20260620 00:12:27.570955 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:27.570955 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:27.570956 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:27.571636 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:27.571913 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:27.573115 5928 hybrid_clock.cc:648] HybridClock initialized: now 1781914347573082 us; error 53 us; skew 500 ppm
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:27.575887 5928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:12:27.576332 5928 webserver.cc:460] Webserver started at http://127.0.98.62:41491/ using document root <none> and password file <none>
I20260620 00:12:27.576650 5928 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:27.576701 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:27.576857 5928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:27.578008 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "d61f21bcfde14cb18da502f3c5a27515"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ea5196c872276ea3c0560774cd636326"
server_key_iv: "fdfdd1a465dc910b6636776bd9018417"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.578393 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "d61f21bcfde14cb18da502f3c5a27515"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ea5196c872276ea3c0560774cd636326"
server_key_iv: "fdfdd1a465dc910b6636776bd9018417"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.579876 5928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:27.580931 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.581313 5928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:27.581418 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "d61f21bcfde14cb18da502f3c5a27515"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ea5196c872276ea3c0560774cd636326"
server_key_iv: "fdfdd1a465dc910b6636776bd9018417"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.581517 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:27.600039 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:27.602527 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:27.602717 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:27.607146 5928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43625
I20260620 00:12:27.607240 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43625 every 8 connection(s)
I20260620 00:12:27.607790 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:27.608670 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:27.611284 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: Bootstrap starting.
I20260620 00:12:27.611917 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:27.611909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 5928
I20260620 00:12:27.612030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:27.612237 392 external_mini_cluster.cc:1428] Setting key c07bbce2580d4489ea7c2d5ee749490c
I20260620 00:12:27.612273 5996 log.cc:824] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:27.613368 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: No bootstrap required, opened a new log
I20260620 00:12:27.616130 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } }
I20260620 00:12:27.616333 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:27.616382 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61f21bcfde14cb18da502f3c5a27515, State: Initialized, Role: FOLLOWER
I20260620 00:12:27.616577 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } }
I20260620 00:12:27.616706 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:27.616766 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:27.616823 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:27.617522 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } }
I20260620 00:12:27.617646 5996 leader_election.cc:302] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d61f21bcfde14cb18da502f3c5a27515; no voters:
I20260620 00:12:27.617848 5996 leader_election.cc:288] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:27.617928 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:27.618129 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:27.618144 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [term 1 LEADER]: Becoming Leader. State: Replica: d61f21bcfde14cb18da502f3c5a27515, State: Running, Role: LEADER
I20260620 00:12:27.618351 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } }
I20260620 00:12:27.618760 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61f21bcfde14cb18da502f3c5a27515. Latest consensus state: current_term: 1 leader_uuid: "d61f21bcfde14cb18da502f3c5a27515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } } }
I20260620 00:12:27.618861 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:27.618757 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d61f21bcfde14cb18da502f3c5a27515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61f21bcfde14cb18da502f3c5a27515" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43625 } } }
I20260620 00:12:27.618968 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515 [sys.catalog]: This master's current role is: LEADER
W20260620 00:12:27.620713 6017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:27.620779 6017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:27.620900 6006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:27.621122 6006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:27.623332 6006 catalog_manager.cc:1347] Generated new cluster ID: 976a80ec225d42b2a88fbf972acdffc8
I20260620 00:12:27.623404 6006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:27.626572 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:27.612845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.0.98.62:43625)
0620 00:12:27.613036 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:27.613046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:27.613083 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:12:27.613748 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:27.613759 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:27.613832 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:27.614021 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:27.614643 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.616318 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.617526 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.618321 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.618825 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:27.621866 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:27.621871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:27.621871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:27.621884 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:27.624285 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:27.624812 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:27.624816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:27.624817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:27.624933 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:27.625170 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:27.625171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:27.625171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:27.625502 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:27.626134 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:27.626214 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:12:27.626361 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260620 00:12:27.635741 6006 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:27.636391 6006 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:27.654865 6006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d61f21bcfde14cb18da502f3c5a27515: Generated new TSK 0
I20260620 00:12:27.655104 6006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:12:27.692853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:27.781977 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:27.782207 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:27.782233 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:27.783447 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:27.783496 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:27.783524 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:27.785202 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:27.785429 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:27.785660 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:27.790820 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:27.790934 6022 server_base.cc:1034] running on GCE node
W20260620 00:12:27.790825 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:27.791023 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:27.791385 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:27.791662 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:27.792829 6022 hybrid_clock.cc:648] HybridClock initialized: now 1781914347792794 us; error 54 us; skew 500 ppm
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:27.795471 6022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:12:27.795966 6022 webserver.cc:460] Webserver started at http://127.0.98.1:39925/ using document root <none> and password file <none>
I20260620 00:12:27.796175 6022 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:27.796216 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:27.796307 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:27.797420 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "a226ddc320fd4095b6a79550132f201b"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "4ba210e61f0d68a45ffb1c753ca62eb6"
server_key_iv: "9b2792c0769b3a3d21434acf7b385caa"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.797724 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a226ddc320fd4095b6a79550132f201b"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "4ba210e61f0d68a45ffb1c753ca62eb6"
server_key_iv: "9b2792c0769b3a3d21434acf7b385caa"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.799031 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:12:27.799752 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.799949 6022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:27.800046 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "a226ddc320fd4095b6a79550132f201b"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "4ba210e61f0d68a45ffb1c753ca62eb6"
server_key_iv: "9b2792c0769b3a3d21434acf7b385caa"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.800125 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:27.825018 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:27.827593 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:27.827728 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:27.828042 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:27.828400 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:27.828437 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.828461 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:27.828474 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.834859 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44149
I20260620 00:12:27.834903 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44149 every 8 connection(s)
I20260620 00:12:27.835278 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:27.838872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6022
I20260620 00:12:27.838997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:27.839210 392 external_mini_cluster.cc:1428] Setting key 61883acc3527428e75d1365f168c049c
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:27.846383 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:27.836159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46977 (local address 127.0.98.62:43625)
0620 00:12:27.836245 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:27.836247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:27.836455 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:12:27.837473 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:27.837474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:27.837485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:27.837550 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:27.838268 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.838769 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.840273 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.840488 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.840648 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:27.843094 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:27.843095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:27.843096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:27.843104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:27.844700 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:27.845243 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:27.845244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:27.845245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:27.845317 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:27.845617 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:27.845618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:27.845618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:27.845797 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:27.845878 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:27.846168 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 00:12:27.846267 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 00:12:27.846472 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:27.836120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43625 (local address 127.0.98.1:46977)
0620 00:12:27.836287 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:27.836333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:27.837359 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:27.837640 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:27.837644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:27.837943 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:27.838160 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:27.838164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.838884 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.838885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:27.840148 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:27.840152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:27.840676 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:27.840678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:27.840842 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:27.841461 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:27.841475 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:27.842974 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:27.844848 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:27.844856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:27.844859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:27.845135 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:27.845438 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:27.845438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:27.845439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:27.845548 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:27.845879 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:27.845884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:27.846026 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 00:12:27.846120 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:12:27.846252 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 00:12:27.847079 6151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43625
I20260620 00:12:27.847200 6151 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:27.847429 6151 heartbeater.cc:511] Master 127.0.98.62:43625 requested a full tablet report, sending...
I20260620 00:12:27.848032 5960 ts_manager.cc:194] Registered new tserver with Master: a226ddc320fd4095b6a79550132f201b (127.0.98.1:44149)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:12:27.848803 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:12:27.872241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:27.958247 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:27.958505 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:27.958532 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:27.959870 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:27.959928 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:27.959959 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:27.961665 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:27.961953 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:27.962217 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:27.967757 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:27.967759 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:27.967757 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:27.967934 6156 server_base.cc:1034] running on GCE node
I20260620 00:12:27.968320 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:27.968659 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:27.969851 6156 hybrid_clock.cc:648] HybridClock initialized: now 1781914347969833 us; error 28 us; skew 500 ppm
Jun 20 00:12:27 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:27.972738 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:12:27.973248 6156 webserver.cc:460] Webserver started at http://127.0.98.2:43113/ using document root <none> and password file <none>
I20260620 00:12:27.973445 6156 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:27.973489 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:27.973582 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:27.974627 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "d93f5f4c22ea4530afa0bbb1b0a364d4"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ac240a30f1b7ddb5822a9283c9860459"
server_key_iv: "5b4faefe26380da1474c5d217ef7bbd4"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.974951 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d93f5f4c22ea4530afa0bbb1b0a364d4"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ac240a30f1b7ddb5822a9283c9860459"
server_key_iv: "5b4faefe26380da1474c5d217ef7bbd4"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.976294 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:27.977257 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.977470 6156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:27.977567 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "d93f5f4c22ea4530afa0bbb1b0a364d4"
format_stamp: "Formatted at 2026-06-20 00:12:27 on dist-test-slave-28bv"
server_key: "ac240a30f1b7ddb5822a9283c9860459"
server_key_iv: "5b4faefe26380da1474c5d217ef7bbd4"
server_key_version: "encryptionkey@0"
I20260620 00:12:27.977630 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:27.992451 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:27.994867 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:27.994992 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:27.995196 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:27.995530 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:27.995584 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:27.995606 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:27.995620 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:28.002737 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40221
I20260620 00:12:28.002777 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40221 every 8 connection(s)
I20260620 00:12:28.003196 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:28.007786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6156
I20260620 00:12:28.007882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:28.008140 392 external_mini_cluster.cc:1428] Setting key 860e201adb9df79fa800b8a9e3ac2e73
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:28.014688 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.004079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34037 (local address 127.0.98.62:43625)
0620 00:12:28.004184 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.004188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.004429 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:12:28.005729 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.005730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.005743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.005830 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.006488 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.007091 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.008227 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.008465 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.008544 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.011445 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:28.011447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:28.011448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:28.011458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:28.013174 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.013679 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.013681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.013681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.013747 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.014017 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.014018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.014018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.014141 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:28.014212 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:28.014502 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 00:12:28.014587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:12:28.014909 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.004054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43625 (local address 127.0.98.2:34037)
0620 00:12:28.004276 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:28.004334 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:28.005578 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:28.005899 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:28.005903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:28.006178 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:28.006384 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:28.006388 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.007188 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.007189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:28.008120 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:28.008123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.008600 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.008601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:28.008723 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.009312 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:28.009324 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:28.011326 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:28.013278 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:28.013286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:28.013288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:28.013591 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:28.013824 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:28.013825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:28.013825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:28.013944 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:28.014213 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:28.014221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:28.014367 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 00:12:28.014482 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:12:28.014654 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260620 00:12:28.015501 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43625
I20260620 00:12:28.015607 6285 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:28.015817 6285 heartbeater.cc:511] Master 127.0.98.62:43625 requested a full tablet report, sending...
I20260620 00:12:28.016250 5960 ts_manager.cc:194] Registered new tserver with Master: d93f5f4c22ea4530afa0bbb1b0a364d4 (127.0.98.2:40221)
I20260620 00:12:28.016736 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34037
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:12:28.041072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:28.132923 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:28.133199 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:28.133239 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:28.134512 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:28.134590 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:28.134641 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:28.136282 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:28.136633 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:28.136896 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:28.142680 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:28.142681 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:28.142709 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:28.142719 6290 server_base.cc:1034] running on GCE node
I20260620 00:12:28.143143 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:28.143383 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:28.144622 6290 hybrid_clock.cc:648] HybridClock initialized: now 1781914348144596 us; error 52 us; skew 500 ppm
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.147641 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:12:28.148159 6290 webserver.cc:460] Webserver started at http://127.0.98.3:33627/ using document root <none> and password file <none>
I20260620 00:12:28.148419 6290 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:28.148492 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:28.148674 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:28.149780 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf62fa5ce1194d58a187589317549732"
format_stamp: "Formatted at 2026-06-20 00:12:28 on dist-test-slave-28bv"
server_key: "02adf7a785800a141b8d40817ad08793"
server_key_iv: "78e870b098a86aa6a073d41fc3f17cb4"
server_key_version: "encryptionkey@0"
I20260620 00:12:28.150264 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf62fa5ce1194d58a187589317549732"
format_stamp: "Formatted at 2026-06-20 00:12:28 on dist-test-slave-28bv"
server_key: "02adf7a785800a141b8d40817ad08793"
server_key_iv: "78e870b098a86aa6a073d41fc3f17cb4"
server_key_version: "encryptionkey@0"
I20260620 00:12:28.152011 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:12:28.153158 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:28.153422 6290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:28.153570 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "bf62fa5ce1194d58a187589317549732"
format_stamp: "Formatted at 2026-06-20 00:12:28 on dist-test-slave-28bv"
server_key: "02adf7a785800a141b8d40817ad08793"
server_key_iv: "78e870b098a86aa6a073d41fc3f17cb4"
server_key_version: "encryptionkey@0"
I20260620 00:12:28.153661 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:28.190935 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:28.193584 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:28.193776 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:28.194042 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:28.194382 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:28.194442 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:28.194485 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:28.194510 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:28.200932 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33207
I20260620 00:12:28.201108 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33207 every 8 connection(s)
I20260620 00:12:28.201397 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:12:28.207451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6290
I20260620 00:12:28.207580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:28.207803 392 external_mini_cluster.cc:1428] Setting key 2887dd8dafaa203e31a76aab50faadb9
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:12:28.212414 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.202276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48445 (local address 127.0.98.62:43625)
0620 00:12:28.202398 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.202401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.202620 (+ 219us) server_negotiation.cc:407] Connection header received
0620 00:12:28.203634 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.203635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.203648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.203701 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.204453 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.205056 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.206107 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.206326 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.206416 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.209235 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:28.209237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:28.209237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:28.209245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:28.210845 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.211327 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.211329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.211329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.211410 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.211690 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.211692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.211692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.211825 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:28.211902 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:28.212212 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 00:12:28.212309 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 00:12:28.212704 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.202184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43625 (local address 127.0.98.3:48445)
0620 00:12:28.202437 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:28.202485 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:28.203518 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:28.203788 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:28.203792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:28.204123 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:28.204361 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:28.204366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.205160 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.205161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:28.205975 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:28.205977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.206438 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.206439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:28.206570 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.207171 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:28.207180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:28.209134 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:28.210965 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:28.210974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:28.210977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:28.211241 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:28.211493 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:28.211494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:28.211494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:28.211614 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:28.211902 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:28.211910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:28.212070 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 00:12:28.212185 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:12:28.212382 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":109,"threads_started":1}
I20260620 00:12:28.213261 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43625
I20260620 00:12:28.213376 6419 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:28.213577 6419 heartbeater.cc:511] Master 127.0.98.62:43625 requested a full tablet report, sending...
I20260620 00:12:28.214046 5960 ts_manager.cc:194] Registered new tserver with Master: bf62fa5ce1194d58a187589317549732 (127.0.98.3:33207)
I20260620 00:12:28.214531 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48445
I20260620 00:12:28.220137 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/test-admin.keytab.
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.239168 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 00:12:28.241312 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:28.240187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.0.98.1:44149)
0620 00:12:28.240432 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.240437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.240448 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:28.240477 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.240479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.240494 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.240669 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.241242 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50932: BlockingRecv error: recv got EOF from 127.0.0.1:50932 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":129,"threads_started":1}
W20260620 00:12:28.241647 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.242815 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.243891 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:28.253585 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.244709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50946 (local address 127.0.98.1:44149)
0620 00:12:28.244771 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.244773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.244798 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:12:28.244874 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.244875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.244886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.244936 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.245367 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.246174 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.246914 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.247269 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.247327 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.249751 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:28.249773 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:28.249773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:28.249781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:28.252078 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.252579 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.252584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.252584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.252674 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.252919 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.252921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.252921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.253091 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:28.253279 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:28.253329 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 00:12:28.253442 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 00:12:28.255990 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:28.255075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.0.98.1:44149)
0620 00:12:28.255130 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.255132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.255189 (+ 57us) server_negotiation.cc:407] Connection header received
0620 00:12:28.255226 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.255227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.255237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.255327 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.255922 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50950: BlockingRecv error: recv got EOF from 127.0.0.1:50950 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 00:12:28.256109 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.257433 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.258495 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:28.267607 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.259254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.0.98.1:44149)
0620 00:12:28.259324 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.259326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.259343 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:12:28.259370 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.259371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.259380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.259441 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.259896 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.260477 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.261693 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.261933 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.262013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.264720 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:28.264722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:28.264722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:28.264732 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:28.266391 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.266846 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.266846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.266847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.266903 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.267117 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.267117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.267118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.267221 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:28.267300 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:28.267403 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:12:28.267476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 00:12:28.269799 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:12:28.269158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.98.1:44149)
0620 00:12:28.269202 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.269204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.269255 (+ 51us) server_negotiation.cc:407] Connection header received
0620 00:12:28.269311 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.269312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.269319 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.269368 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.269724 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50980: BlockingRecv error: recv got EOF from 127.0.0.1:50980 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 00:12:28.269976 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914322045185-392-0/krb5kdc/krb5cc)
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.271171 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:28.272328 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:12:28.281153 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:28.273174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.0.98.1:44149)
0620 00:12:28.273241 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:28.273243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:28.273253 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:28.273288 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:28.273289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:28.273297 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:28.273364 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:28.273859 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.274354 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.275191 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:28.275521 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:28.275592 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:28.278322 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:28.278324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:28.278325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:28.278334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:28.279750 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.280185 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.280186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.280187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.280253 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:28.280476 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:28.280477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:28.280477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:28.280672 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:28.280799 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:28.280925 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 00:12:28.281036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:12:28.282032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6022
I20260620 00:12:28.288725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6156
I20260620 00:12:28.293944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6290
I20260620 00:12:28.298861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 5928
2026-06-20T00:12:28Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2065 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[6439](info): set up 2 sockets
Jun 20 00:12:28 dist-test-slave-28bv krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260620 00:12:29.372932 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:12:29 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:12:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:12:29.509160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:29.599681 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:29.599985 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:29.600013 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:29.601465 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:29.601536 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:29.601552 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:29.601565 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:29.601578 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:29.603327 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:29.603668 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:29.604001 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:29.609580 6452 server_base.cc:1034] running on GCE node
W20260620 00:12:29.609539 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:29.609534 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:29.609545 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:29.610036 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:29.610396 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:29.611610 6452 hybrid_clock.cc:648] HybridClock initialized: now 1781914349611583 us; error 41 us; skew 500 ppm
Jun 20 00:12:29 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:29.614987 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:12:29.615638 6452 webserver.cc:460] Webserver started at http://127.0.98.62:34467/ using document root <none> and password file <none>
I20260620 00:12:29.615934 6452 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:29.615999 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:29.616143 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:29.617523 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "6ace22bf7b20425e973c73cde979f0f7"
format_stamp: "Formatted at 2026-06-20 00:12:29 on dist-test-slave-28bv"
server_key: "cc97ac8de4243d93a6b9b450b0408a5b"
server_key_iv: "fb000d2bffc765cd1eb0bd450493ab83"
server_key_version: "encryptionkey@0"
I20260620 00:12:29.617945 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ace22bf7b20425e973c73cde979f0f7"
format_stamp: "Formatted at 2026-06-20 00:12:29 on dist-test-slave-28bv"
server_key: "cc97ac8de4243d93a6b9b450b0408a5b"
server_key_iv: "fb000d2bffc765cd1eb0bd450493ab83"
server_key_version: "encryptionkey@0"
I20260620 00:12:29.619678 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:12:29.620669 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:29.620914 6452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:29.621032 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "6ace22bf7b20425e973c73cde979f0f7"
format_stamp: "Formatted at 2026-06-20 00:12:29 on dist-test-slave-28bv"
server_key: "cc97ac8de4243d93a6b9b450b0408a5b"
server_key_iv: "fb000d2bffc765cd1eb0bd450493ab83"
server_key_version: "encryptionkey@0"
I20260620 00:12:29.621140 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:29.656886 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:29.659363 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:29.659543 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:29.664117 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33771
I20260620 00:12:29.664158 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33771 every 8 connection(s)
I20260620 00:12:29.664601 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:12:29.665221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6452
I20260620 00:12:29.665637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:12:29.665874 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:29.665905 392 external_mini_cluster.cc:1428] Setting key e6bd86a7ce0e17b98c939e7a9a6aa071
I20260620 00:12:29.669431 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Bootstrap starting.
I20260620 00:12:29.670285 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:29.670612 6520 log.cc:824] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:29.671460 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: No bootstrap required, opened a new log
Jun 20 00:12:29 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:12:29.674345 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } }
I20260620 00:12:29.674558 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:29.674633 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ace22bf7b20425e973c73cde979f0f7, State: Initialized, Role: FOLLOWER
I20260620 00:12:29.674817 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } }
I20260620 00:12:29.674916 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:12:29.674955 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:12:29.675004 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:29.678047 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:29.666598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.0.98.62:33771)
0620 00:12:29.666809 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:29.666819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:29.666854 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:12:29.667378 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:29.667386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:29.667439 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:29.667586 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:29.668474 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:29.669694 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:29.670836 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:29.671432 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:29.671525 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:29.674442 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:29.674460 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:29.674461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:29.674474 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:29.676459 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:29.676916 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:29.676919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:29.676919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:29.677014 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:29.677196 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:29.677197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:29.677197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:29.677443 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:29.677552 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:29.677642 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:12:29.677838 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260620 00:12:29.699818 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } }
I20260620 00:12:29.700038 6520 leader_election.cc:302] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ace22bf7b20425e973c73cde979f0f7; no voters:
I20260620 00:12:29.700281 6520 leader_election.cc:288] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:12:29.700408 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:29.700618 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:29.700683 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6ace22bf7b20425e973c73cde979f0f7, State: Running, Role: LEADER
I20260620 00:12:29.700853 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } }
I20260620 00:12:29.701324 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } }
I20260620 00:12:29.701416 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:29.701599 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ace22bf7b20425e973c73cde979f0f7. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } }
I20260620 00:12:29.701660 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
W20260620 00:12:29.702598 6540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:29.702661 6540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:29.702750 6534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:12:29.702915 6534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:12:29.705333 6534 catalog_manager.cc:1347] Generated new cluster ID: 175ef38ffff9424f8354a31b4af60900
I20260620 00:12:29.705397 6534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:12:29.717121 6534 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:12:29.717881 6534 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:12:29.721992 6534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Generated new TSK 0
I20260620 00:12:29.722353 6534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
W20260620 00:12:29.975924 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.1: real 0.245s user 0.000s sys 0.003s
I20260620 00:12:29.976387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:30.063387 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:30.063684 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:30.063730 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:30.065196 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:30.065299 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:30.065351 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:30.067066 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:30.067400 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:30.067667 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:30.072906 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.072957 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.072932 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.073148 6546 server_base.cc:1034] running on GCE node
I20260620 00:12:30.073372 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.073647 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.074801 6546 hybrid_clock.cc:648] HybridClock initialized: now 1781914350074777 us; error 37 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.078243 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:12:30.078826 6546 webserver.cc:460] Webserver started at http://127.0.98.1:39493/ using document root <none> and password file <none>
I20260620 00:12:30.079089 6546 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.079166 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.079329 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:30.080529 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "c2a44fa171554e048562cfed9e9ad5d6"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "867ef330eb79a0b91ddc59461cfaa3ef"
server_key_iv: "b6fe5231d29da581873c99b5c5947e9c"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.081004 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c2a44fa171554e048562cfed9e9ad5d6"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "867ef330eb79a0b91ddc59461cfaa3ef"
server_key_iv: "b6fe5231d29da581873c99b5c5947e9c"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.082547 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:12:30.083514 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.083767 6546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:30.083884 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "c2a44fa171554e048562cfed9e9ad5d6"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "867ef330eb79a0b91ddc59461cfaa3ef"
server_key_iv: "b6fe5231d29da581873c99b5c5947e9c"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.083981 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.100483 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.103178 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:30.103363 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.103631 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:30.103966 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:30.104022 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.104064 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:30.104108 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.110576 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39081
I20260620 00:12:30.110617 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39081 every 8 connection(s)
I20260620 00:12:30.110965 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:30.111716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6546
I20260620 00:12:30.111830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:12:30.112052 392 external_mini_cluster.cc:1428] Setting key ac54d91ac1538a9337f6736c36d089c5
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:12:30.122618 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.111700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55419 (local address 127.0.98.62:33771)
0620 00:12:30.111783 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.111785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.112006 (+ 221us) server_negotiation.cc:407] Connection header received
0620 00:12:30.112622 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.112623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.112634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.112684 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.113703 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.114462 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.116111 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.116455 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.116526 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.119295 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.119297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.119297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.119304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.121120 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.121540 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.121541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.121541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.121604 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.121897 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.121898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.121898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.122027 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.122079 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.122336 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.122473 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:12:30.122845 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.111656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:55419)
0620 00:12:30.111851 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.111896 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.112479 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.112781 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.112793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.113397 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.113597 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.113602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.114584 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.114586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.116016 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.116020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.116594 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.116596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.116768 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.117315 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.117327 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.119166 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.121202 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.121209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.121211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.121436 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.121671 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.121671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.121672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.121844 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.122084 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.122089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.122219 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.122308 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.122481 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260620 00:12:30.123400 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260620 00:12:30.123530 6675 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:30.123744 6675 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:30.124326 6484 ts_manager.cc:194] Registered new tserver with Master: c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081)
I20260620 00:12:30.125208 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:12:30.139890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:30.228354 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:30.228650 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:30.228681 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:30.229882 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:30.229925 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:30.229979 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:30.231506 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:30.231766 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:30.232007 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:30.237433 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.237426 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.237490 6680 server_base.cc:1034] running on GCE node
W20260620 00:12:30.237426 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.237885 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.238154 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.239336 6680 hybrid_clock.cc:648] HybridClock initialized: now 1781914350239327 us; error 37 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.242364 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:12:30.242960 6680 webserver.cc:460] Webserver started at http://127.0.98.2:46213/ using document root <none> and password file <none>
I20260620 00:12:30.243207 6680 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.243278 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.243449 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:30.244658 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "5159394615d747c39936d0bdf6fa4737"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "528faa9874ff4ac34577825053495197"
server_key_iv: "1ca6a94351aeb604015b40da83af263f"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.245039 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5159394615d747c39936d0bdf6fa4737"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "528faa9874ff4ac34577825053495197"
server_key_iv: "1ca6a94351aeb604015b40da83af263f"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.246477 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:12:30.247264 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.247494 6680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:30.247603 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "5159394615d747c39936d0bdf6fa4737"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "528faa9874ff4ac34577825053495197"
server_key_iv: "1ca6a94351aeb604015b40da83af263f"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.247692 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.257541 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.259733 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:30.259905 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.260157 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:30.260485 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:30.260542 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.260636 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:30.260664 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.267283 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45601
I20260620 00:12:30.267340 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45601 every 8 connection(s)
I20260620 00:12:30.267681 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:12:30.277578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6680
I20260620 00:12:30.277701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:12:30.277882 392 external_mini_cluster.cc:1428] Setting key 78a580b25ed560e96f5da87a79637bbd
I20260620 00:12:30.277906 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.268458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36109 (local address 127.0.98.62:33771)
0620 00:12:30.268508 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.268510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.268728 (+ 218us) server_negotiation.cc:407] Connection header received
0620 00:12:30.269297 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.269298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.269312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.269364 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.270405 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.270909 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.271968 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.272177 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.272260 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.274862 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.274864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.274865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.274877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.276342 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.276875 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.276876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.276877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.276938 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.277214 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.277214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.277215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.277339 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.277413 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.277711 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.277802 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:12:30.278101 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.268400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:36109)
0620 00:12:30.268599 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.268646 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.269171 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.269437 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.269441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.270081 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.270281 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.270286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.271009 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.271011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.271864 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.271866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.272268 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.272269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.272419 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.273064 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.273073 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.274745 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.276469 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.276477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.276479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.276790 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.277013 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.277014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.277014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.277139 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.277416 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.277423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.277573 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.277677 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.277847 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260620 00:12:30.278623 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
I20260620 00:12:30.278730 6809 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:30.278919 6809 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:30.279366 6484 ts_manager.cc:194] Registered new tserver with Master: 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
I20260620 00:12:30.279798 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36109
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:12:30.307601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:30.390455 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:30.390681 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:30.390703 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:30.391782 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:30.391817 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:30.391844 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:30.393258 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:30.393441 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:30.393626 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:30.399333 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.399362 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.399355 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.399386 6814 server_base.cc:1034] running on GCE node
I20260620 00:12:30.399804 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.400135 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.401270 6814 hybrid_clock.cc:648] HybridClock initialized: now 1781914350401248 us; error 37 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.403915 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:12:30.404356 6814 webserver.cc:460] Webserver started at http://127.0.98.3:37177/ using document root <none> and password file <none>
I20260620 00:12:30.404640 6814 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.404700 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.404830 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:30.405871 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "aaf420f438df4751b5d23ce4f5d398e4"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "e2972f03ce085296edea02da70f47518"
server_key_iv: "b2318865a7826a12ec59f9a48aface78"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.406258 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aaf420f438df4751b5d23ce4f5d398e4"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "e2972f03ce085296edea02da70f47518"
server_key_iv: "b2318865a7826a12ec59f9a48aface78"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.408054 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:12:30.408910 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.409111 6814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:30.409219 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "aaf420f438df4751b5d23ce4f5d398e4"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "e2972f03ce085296edea02da70f47518"
server_key_iv: "b2318865a7826a12ec59f9a48aface78"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.409282 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.426199 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.428514 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:30.428781 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.429035 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:30.429378 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:30.429435 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.429476 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:30.429528 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.435592 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45307
I20260620 00:12:30.435636 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45307 every 8 connection(s)
I20260620 00:12:30.435966 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:12:30.443226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6814
I20260620 00:12:30.443336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:12:30.443548 392 external_mini_cluster.cc:1428] Setting key c8bd0529e42278bcc7c028f05ade5f32
I20260620 00:12:30.445567 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.436790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35431 (local address 127.0.98.62:33771)
0620 00:12:30.436891 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.436894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.437057 (+ 163us) server_negotiation.cc:407] Connection header received
0620 00:12:30.437632 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.437634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.437647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.437766 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.438747 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.439260 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.440256 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.440473 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.440598 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.442668 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.442670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.442671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.442679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.444291 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.444748 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.444749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.444749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.444804 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.445037 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.445037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.445038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.445136 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.445183 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.445430 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.445497 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 00:12:30.445714 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.436701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:35431)
0620 00:12:30.436916 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.436967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.437482 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.437855 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.437858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.438440 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.438644 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.438648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.439357 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.439358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.440158 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.440160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.440591 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.440592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.440723 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.441211 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.441219 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.442564 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.444390 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.444396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.444397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.444695 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.444865 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.444865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.444865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.444967 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.445187 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.445191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.445309 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.445386 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.445515 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260620 00:12:30.446208 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
I20260620 00:12:30.446322 6943 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:30.446525 6943 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:30.446913 6484 ts_manager.cc:194] Registered new tserver with Master: aaf420f438df4751b5d23ce4f5d398e4 (127.0.98.3:45307)
I20260620 00:12:30.447301 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35431
I20260620 00:12:30.455403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 00:12:30.483963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43293
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33771,127.0.98.61:43293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:30.567050 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:30.567346 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:30.567391 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:30.568912 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:30.569020 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:30.569049 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:30.569070 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:30.569088 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:30.571074 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33771,127.0.98.61:43293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
W20260620 00:12:30.571249 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33771,127.0.98.61:43293'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 00:12:30.571439 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:30.571717 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:30.577057 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.577059 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.577075 6948 server_base.cc:1034] running on GCE node
W20260620 00:12:30.577057 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.577636 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.577917 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.579063 6948 hybrid_clock.cc:648] HybridClock initialized: now 1781914350579050 us; error 29 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.581786 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:12:30.582316 6948 webserver.cc:460] Webserver started at http://127.0.98.61:43857/ using document root <none> and password file <none>
I20260620 00:12:30.582568 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.582614 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.582747 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:30.583935 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/instance:
uuid: "f8cc7382290d4f10bec0f8e0917a5979"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "69c0bffc2697860b95e1de211705a1ca"
server_key_iv: "81bdb5961ec8ab3a3fd8b416581cca43"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.584313 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal/instance:
uuid: "f8cc7382290d4f10bec0f8e0917a5979"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "69c0bffc2697860b95e1de211705a1ca"
server_key_iv: "81bdb5961ec8ab3a3fd8b416581cca43"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.585759 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:30.586546 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.586774 6948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:30.586879 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "f8cc7382290d4f10bec0f8e0917a5979"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "69c0bffc2697860b95e1de211705a1ca"
server_key_iv: "81bdb5961ec8ab3a3fd8b416581cca43"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.586978 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.606648 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.609025 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:30.609203 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.613315 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43293
I20260620 00:12:30.613350 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43293 every 8 connection(s)
I20260620 00:12:30.613750 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:12:30.614791 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:30.616415 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:12:30.619536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 6948
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:30.627019 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.617334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.0.98.62:33771)
0620 00:12:30.617403 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.617406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.617702 (+ 296us) server_negotiation.cc:407] Connection header received
0620 00:12:30.618120 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.618121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.618132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.618191 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.618783 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.619268 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.620681 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.620870 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.620925 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.623555 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.623556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.623557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.623565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.625338 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.625855 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.625857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.625857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.625921 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.626166 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.626167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.626168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.626324 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.626421 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.626798 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.626911 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:12:30.627130 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.617329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56118)
0620 00:12:30.617511 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.617551 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.618039 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.618244 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.618247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.618500 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.618709 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.618713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.619379 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.619380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.620589 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.620592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.620952 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.620954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.621109 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.621698 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.621711 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.623384 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.625437 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.625443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.625445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.625776 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.625991 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.625991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.625992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.626092 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.626431 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.626436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.626601 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.626742 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.626911 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260620 00:12:30.627717 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:12:30.629522 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.620125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.0.98.61:43293)
0620 00:12:30.620277 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.620282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.620301 (+ 19us) server_negotiation.cc:407] Connection header received
0620 00:12:30.620357 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.620358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.620374 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.620436 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.621194 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.621880 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.622746 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.623213 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.623269 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.626145 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.626160 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.626160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.626167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.628243 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.628636 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.628639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.628640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.628704 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.628911 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.628912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.628912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.629050 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.629239 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.629291 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.629412 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:30.635115 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.628080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.0.98.61:43293)
0620 00:12:30.628239 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.628240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.628246 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:12:30.628260 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.628260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.628267 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.628342 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.628679 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.629163 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.630076 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.630351 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.630419 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.632190 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.632192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.632192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.632198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.634131 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.634425 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.634426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.634426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.634475 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.634668 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.634669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.634669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.634769 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.634821 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.634938 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.635015 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260620 00:12:30.635167 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.628030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52088)
0620 00:12:30.628073 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.628082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.628119 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.628344 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.628345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.628480 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.628521 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.628522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.629250 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.629251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.629972 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.629974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.630418 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.630418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.630473 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.630904 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.630909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.632112 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.634205 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.634206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.634206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.634361 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.634529 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.634530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.634530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.634578 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.634823 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.634824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.634859 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.634923 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.635076 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":69}
I20260620 00:12:30.636679 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Bootstrap starting.
I20260620 00:12:30.637331 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:30.637610 7016 log.cc:824] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:30.638396 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: No bootstrap required, opened a new log
I20260620 00:12:30.639807 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } }
I20260620 00:12:30.639933 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:30.639956 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cc7382290d4f10bec0f8e0917a5979, State: Initialized, Role: FOLLOWER
I20260620 00:12:30.640115 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } }
I20260620 00:12:30.640453 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:30.640442 7023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } } }
I20260620 00:12:30.640637 7023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:12:30.642373 7034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:30.642438 7034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:30.644187 6948 master_runner.cc:428] Detected that this master f8cc7382290d4f10bec0f8e0917a5979 is joining an existing cluster
I20260620 00:12:30.644253 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43293
I20260620 00:12:30.653108 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.645335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.0.98.62:33771)
0620 00:12:30.645377 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.645379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.645710 (+ 331us) server_negotiation.cc:407] Connection header received
0620 00:12:30.645763 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.645764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.645775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.645830 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.646412 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.646913 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.647979 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.648229 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.648262 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.649922 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.649923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.649924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.649930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.651740 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652182 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.652183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.652184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.652245 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652463 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.652463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.652464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.652659 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.652706 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.652954 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.653041 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:30.653446 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.645265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56126)
0620 00:12:30.645631 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.645641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.645702 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.645891 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.645891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.646207 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.646311 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.646313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.646997 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.646999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.647881 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.647883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.648346 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.648346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.648387 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.649139 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.649145 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.649810 (+ 665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.651845 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.651846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.651847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.652096 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.652308 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.652309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.652379 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.652751 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.652751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.652829 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.653228 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.653346 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260620 00:12:30.653764 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.645551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.0.98.61:43293)
0620 00:12:30.645590 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.645592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.645733 (+ 141us) server_negotiation.cc:407] Connection header received
0620 00:12:30.645815 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.645815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.645824 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.645870 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.646395 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.647130 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.648084 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.648406 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.648493 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.649906 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.649908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.649908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.649915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.652314 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652702 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.652703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.652703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.652764 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652964 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.652965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.652965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.653092 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.653169 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.653634 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.653712 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:12:30.654040 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.645505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52104)
0620 00:12:30.645637 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.645647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.645756 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.645889 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.645890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.646204 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.646311 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.646313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.647230 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.647231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.647997 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.647998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.648501 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.648502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.648615 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.649154 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.649159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.649810 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.652392 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.652393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.652628 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.652836 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.652836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.652837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.652900 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.653455 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.653456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.653500 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.653866 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.653954 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":16,"thread_start_us":65,"threads_started":1}
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:30.664377 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.656134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.61:43293)
0620 00:12:30.656173 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.656175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.656322 (+ 147us) server_negotiation.cc:407] Connection header received
0620 00:12:30.656375 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.656376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.656387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.656452 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.656985 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.657727 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.658500 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.658800 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.658872 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.661272 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.661274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.661274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.661282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.662920 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.663333 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.663335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.663335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.663397 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.663624 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.663625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.663625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.663792 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.663866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.664182 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.664275 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:12:30.664390 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.656050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52116)
0620 00:12:30.656212 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.656254 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.656317 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.656455 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.656458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.656855 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.656908 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.656910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.657816 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.657817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.658417 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.658419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.658896 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.658897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.658967 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.659495 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.659500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.661181 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.663016 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.663024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.663026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.663255 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.663486 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.663487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.663487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.663553 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.663883 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.663891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.663943 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.664146 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.664280 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260620 00:12:30.665525 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43293: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43293
}
attrs {
promote: true
}
}
dest_uuid: "6ace22bf7b20425e973c73cde979f0f7"
cas_config_opid_index: -1
I20260620 00:12:30.665861 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } }
I20260620 00:12:30.666707 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f8cc7382290d4f10bec0f8e0917a5979 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } }
I20260620 00:12:30.667071 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:30.667829 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43293
I20260620 00:12:30.667881 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Refusing update from remote peer 6ace22bf7b20425e973c73cde979f0f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 00:12:30.668126 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } }
I20260620 00:12:30.668220 6948 master.cc:537] Master@127.0.98.61:43293 shutting down...
I20260620 00:12:30.668310 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:30.668429 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:12:30.669145 6468 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 00:12:30.669613 6468 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 -> Peer f8cc7382290d4f10bec0f8e0917a5979 (127.0.98.61:43293): Couldn't send request to peer f8cc7382290d4f10bec0f8e0917a5979. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:12:30.670794 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:30.670888 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:30.670931 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: stopping tablet replica
I20260620 00:12:30.684274 6948 master.cc:552] Master@127.0.98.61:43293 shutdown complete.
I20260620 00:12:30.684352 6948 master_runner.cc:305] Clearing existing system tablet
I20260620 00:12:30.684504 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.684595 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.685273 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:12:30.685766 7048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.685937 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.686030 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:12:30.686062 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "f8cc7382290d4f10bec0f8e0917a5979"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "69c0bffc2697860b95e1de211705a1ca"
server_key_iv: "81bdb5961ec8ab3a3fd8b416581cca43"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.687029 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:30.688766 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:12:30.688860 6948 log.cc:1196] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 00:12:30.689081 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Deleting consensus metadata
I20260620 00:12:30.689244 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:33771
I20260620 00:12:30.689941 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33771
I20260620 00:12:30.697821 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.690387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.0.98.62:33771)
0620 00:12:30.690450 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.690453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.690537 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:12:30.690572 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.690573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.690585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.690675 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.691165 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.692031 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.692971 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.693310 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.693400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.694545 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.694547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.694547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.694556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.696250 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.696873 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.696875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.696875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.696948 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.697196 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.697198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.697198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.697349 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.697425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.697631 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.697720 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:12:30.697837 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.690310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56128)
0620 00:12:30.690447 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.690460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.690526 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.690724 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.690733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.690985 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.691066 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.691068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.692145 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.692146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.692863 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.692865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.693475 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.693476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.693533 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.694102 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.694107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.694444 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.696379 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.696380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.696381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.696784 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.697060 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.697061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.697061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.697121 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.697442 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.697443 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.697500 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.697578 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.697751 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":96,"threads_started":1}
I20260620 00:12:30.698558 6504 tablet_copy_service.cc:140] P 6ace22bf7b20425e973c73cde979f0f7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f8cc7382290d4f10bec0f8e0917a5979 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56128)
I20260620 00:12:30.698670 6504 tablet_copy_service.cc:161] P 6ace22bf7b20425e973c73cde979f0f7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f8cc7382290d4f10bec0f8e0917a5979 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56128: session id = f8cc7382290d4f10bec0f8e0917a5979-00000000000000000000000000000000
I20260620 00:12:30.699331 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:12:30.700018 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:12:30.701454 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: tablet copy: Starting download of 0 data blocks...
I20260620 00:12:30.701644 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: tablet copy: Starting download of 1 WAL segments...
I20260620 00:12:30.702696 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:12:30.703694 6504 tablet_copy_service.cc:342] P 6ace22bf7b20425e973c73cde979f0f7: Request end of tablet copy session f8cc7382290d4f10bec0f8e0917a5979-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56128
I20260620 00:12:30.703776 6504 tablet_copy_service.cc:433] P 6ace22bf7b20425e973c73cde979f0f7: ending tablet copy session f8cc7382290d4f10bec0f8e0917a5979-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f8cc7382290d4f10bec0f8e0917a5979
I20260620 00:12:30.706696 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:30.707994 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.708088 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.708184 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.708385 6948 server_base.cc:1034] running on GCE node
I20260620 00:12:30.708482 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.708678 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.709837 6948 hybrid_clock.cc:648] HybridClock initialized: now 1781914350709829 us; error 25 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.711347 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:12:30.711725 6948 webserver.cc:460] Webserver started at http://127.0.98.61:45431/ using document root <none> and password file <none>
I20260620 00:12:30.711863 6948 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.711910 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.712707 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:30.713238 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.713474 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:12:30.713560 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "f8cc7382290d4f10bec0f8e0917a5979"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "69c0bffc2697860b95e1de211705a1ca"
server_key_iv: "81bdb5961ec8ab3a3fd8b416581cca43"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.713627 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.740399 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.742276 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.745533 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43293
I20260620 00:12:30.745548 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43293 every 8 connection(s)
I20260620 00:12:30.745925 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:12:30.746611 7118 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:12:30.747047 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Bootstrap starting.
I20260620 00:12:30.749408 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:12:30.749655 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: Bootstrap complete.
I20260620 00:12:30.749864 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } }
I20260620 00:12:30.749936 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8cc7382290d4f10bec0f8e0917a5979, State: Initialized, Role: LEARNER
I20260620 00:12:30.749962 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } }
I20260620 00:12:30.750167 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:30.750168 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } }
I20260620 00:12:30.750344 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:12:30.751693 7132 catalog_manager.cc:1259] Loaded cluster ID: 175ef38ffff9424f8354a31b4af60900
I20260620 00:12:30.751801 7132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: loading cluster ID for follower catalog manager: success
I20260620 00:12:30.752959 7132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: acquiring CA information for follower catalog manager: success
I20260620 00:12:30.753314 7132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:12:30.789892 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.782180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.0.98.61:43293)
0620 00:12:30.782324 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.782326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.782335 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:30.782359 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.782359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.782369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.782441 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.783207 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.783880 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.785061 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.785264 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.785325 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.786815 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.786816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.786816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.786823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.788553 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.788985 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.788987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.788987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.789048 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.789278 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.789280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.789280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.789404 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.789489 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.789609 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.789729 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 00:12:30.821206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45695
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:12:30.906193 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:30.906452 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:30.906478 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:30.907692 7136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:12:30.907753 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:30.907775 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:30.907830 7136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:12:30.907848 7136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:12:30.909618 7136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:30.909935 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:30.910202 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:12:30.915670 7136 server_base.cc:1034] running on GCE node
W20260620 00:12:30.915652 7141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.915630 7144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:30.915630 7142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:30.916322 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:30.916736 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:30.917936 7136 hybrid_clock.cc:648] HybridClock initialized: now 1781914350917951 us; error 59 us; skew 500 ppm
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:30.921353 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:12:30.922004 7136 webserver.cc:460] Webserver started at http://127.0.98.60:39385/ using document root <none> and password file <none>
I20260620 00:12:30.922267 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:30.922339 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:30.922493 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:12:30.923616 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/instance:
uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "c85ea7aa16b9b7047f09dc2ec8dfd580"
server_key_iv: "a60386010fe7aee2affa2dc0239eb03e"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.924007 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal/instance:
uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "c85ea7aa16b9b7047f09dc2ec8dfd580"
server_key_iv: "a60386010fe7aee2affa2dc0239eb03e"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.925585 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:12:30.926519 7151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:30.926745 7136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:30.926870 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "c85ea7aa16b9b7047f09dc2ec8dfd580"
server_key_iv: "a60386010fe7aee2affa2dc0239eb03e"
server_key_version: "encryptionkey@0"
I20260620 00:12:30.926954 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:30.941574 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:30.944259 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:30.944461 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:30.948976 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45695
I20260620 00:12:30.949126 7203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45695 every 8 connection(s)
I20260620 00:12:30.949460 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:12:30.950856 7204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:30.952756 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:12:30.956611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 7136
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:12:30.964390 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.953707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.0.98.62:33771)
0620 00:12:30.953822 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.953825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.954037 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:12:30.954579 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.954580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.954594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.954660 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.955774 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.956284 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.957783 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.958009 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.958079 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.960914 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.960916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.960916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.960924 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.962753 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.963328 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.963330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.963331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.963402 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.963684 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.963685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.963685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.963817 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.963900 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.964205 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.964294 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 00:12:30.964645 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.953582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56136)
0620 00:12:30.953896 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.953936 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.954462 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.954741 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.954745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.955011 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.955646 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.955651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.956422 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.956423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.957674 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.957677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.958106 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.958107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.958261 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.958859 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.958868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.960754 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.962863 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.962871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.962874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.963225 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.963482 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.963483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.963483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.963604 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.963890 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.963896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.964046 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.964160 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.964331 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260620 00:12:30.965384 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:12:30.966719 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.957253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.0.98.60:45695)
0620 00:12:30.957428 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.957437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.957452 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:12:30.957471 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.957473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.957489 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.957563 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.958311 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.959164 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.960019 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.960590 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.960666 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.963422 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.963432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.963432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.963440 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.965302 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.965766 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.965770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.965770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.965835 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.966060 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.966061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.966061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.966275 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.966438 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.966487 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.966598 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":115,"threads_started":1}
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:30.973414 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.966182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52156)
0620 00:12:30.966245 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.966256 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.966324 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.966485 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.966486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.966643 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.966741 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.966743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.967619 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.967620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.968145 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.968146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.968572 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.968573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.968645 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.969081 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.969086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.970477 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.972244 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.972245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.972246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.972426 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.972721 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.972722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.972722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.972786 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.973055 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.973056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.973103 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.973201 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.973313 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:12:30.973414 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.966272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.61:43293)
0620 00:12:30.966346 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.966349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.966370 (+ 21us) server_negotiation.cc:407] Connection header received
0620 00:12:30.966397 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.966398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.966411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.966482 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.966809 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.967533 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.968241 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.968466 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.968573 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.970572 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.970574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.970574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.970582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.972125 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.972519 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.972521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.972521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.972622 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.972851 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.972852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.972852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.972976 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.973051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.973230 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.973316 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:12:30.974066 7204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45695 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:12:30 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914350, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:12:30.980895 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.974369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.0.1:39142)
0620 00:12:30.974400 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.974410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.974445 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.974549 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.974549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.974663 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.974698 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.974700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.975380 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.975381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.975890 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.975892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.976328 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.976330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.976374 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.976898 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.976903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.978178 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.979876 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.979877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.979877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.980042 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.980265 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.980265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.980265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.980316 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.980582 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:30.980583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:30.980633 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:12:30.980688 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:12:30.980821 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 00:12:30.981009 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.974406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.0.98.60:45695)
0620 00:12:30.974455 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.974457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.974471 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:12:30.974487 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.974487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.974496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.974545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.974806 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.975289 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.975996 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.976221 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.976302 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.978269 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.978270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.978271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.978277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.979769 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.980138 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.980139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.980139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.980194 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.980378 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.980379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.980379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.980486 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:30.980600 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:30.980712 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 00:12:30.980885 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":29}
I20260620 00:12:30.982328 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Bootstrap starting.
I20260620 00:12:30.983124 7204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:30.983361 7204 log.cc:824] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:30.984113 7204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: No bootstrap required, opened a new log
I20260620 00:12:30.985746 7204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45695 } }
I20260620 00:12:30.985910 7204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:30.985935 7204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9a5161bdfd4b35a4b85c50546a0a03, State: Initialized, Role: FOLLOWER
I20260620 00:12:30.986069 7204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45695 } }
I20260620 00:12:30.986330 7209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45695 } } }
I20260620 00:12:30.986408 7204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:30.986413 7209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:12:30.988909 7220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:12:30.989001 7220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:12:30.991544 7136 master_runner.cc:428] Detected that this master 3f9a5161bdfd4b35a4b85c50546a0a03 is joining an existing cluster
I20260620 00:12:30.991614 7136 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:45695
I20260620 00:12:31.000453 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.992867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.0.98.61:43293)
0620 00:12:30.992913 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.992915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.993084 (+ 169us) server_negotiation.cc:407] Connection header received
0620 00:12:30.993196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.993197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.993208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.993265 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.993732 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.994510 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.995664 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.995960 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996005 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.997327 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.997328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.997328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.997336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:30.999153 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.999555 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.999556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.999558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.999622 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:30.999854 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:30.999855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:30.999855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:30.999997 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.000073 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.000242 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.000336 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:12:31.000479 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.992814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52170)
0620 00:12:30.993018 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.993031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.993113 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.993318 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.993320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.993545 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.993636 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.993638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.994614 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.994615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.995566 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.995569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.996097 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.996162 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.996731 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.996736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.997210 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:30.999241 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.999243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.999244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.999468 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:30.999702 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:30.999703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:30.999703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:30.999780 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.000075 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.000076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.000123 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.000207 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.000355 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260620 00:12:31.001709 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.993639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.0.98.62:33771)
0620 00:12:30.993724 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.993726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.993734 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:12:30.993758 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.993759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.993771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.993828 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.994725 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.995494 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996571 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.996874 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996917 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.998642 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:30.998643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:30.998644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:30.998651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.000323 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.000806 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.000807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.000808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.000873 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.001090 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.001091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.001092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.001241 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.001344 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.001528 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.001616 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:12:31.001750 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.992839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56146)
0620 00:12:30.993234 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.993244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.993386 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.993873 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.993873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.994121 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.994619 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.994621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.995581 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.995582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.996441 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.996444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.996990 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.997055 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:30.998118 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:30.998123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:30.998530 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.000406 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.000407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.000407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.000722 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.000950 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.000951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.000951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.001019 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.001375 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.001376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.001418 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.001483 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.001587 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914349, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:12:31.018317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6546
I20260620 00:12:31.024356 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.004108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.0.1:39158)
0620 00:12:31.004167 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.004179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.004247 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.004521 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.004522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.004736 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.004781 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.004783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.005740 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.005741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.006270 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.006272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.006777 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.006778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.006837 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.007359 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.007363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.008733 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.023086 (+ 14353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.023088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.023350 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.023585 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.023586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.023652 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.023982 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.023983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.024035 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.024085 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.024234 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:12:31.024449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39081
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33771
--webserver_port=39493
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:12:31.024883 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.004190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.0.98.60:45695)
0620 00:12:31.004378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.004380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.004388 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:12:31.004424 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.004425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.004439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.004512 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.004856 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.005651 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.006350 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.006683 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.006756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.008832 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.008833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.008834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.008841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.022978 (+ 14137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023439 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.023440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.023441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.023515 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023734 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.023735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.023735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.023891 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.024580 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.024625 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.024714 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260620 00:12:31.025667 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.993638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.0.98.60:45695)
0620 00:12:30.993686 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:30.993688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:30.993759 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:12:30.993854 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:30.993855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:30.993863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:30.993889 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:30.994714 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.995483 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.996933 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.997465 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.997510 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.021941 (+ 24431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.021943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.021944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.021951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.023602 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023963 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.023964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.023964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.024019 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.024190 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.024190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.024191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.025146 (+ 955us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.025197 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.025405 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.025545 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":27}
I20260620 00:12:31.025573 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:30.993108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.0.1:39156)
0620 00:12:30.993699 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:30.993709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:30.993779 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:30.993940 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:30.993941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:30.994149 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:30.994619 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.994621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.995566 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.995567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.996847 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:30.996850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:30.997581 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:30.997582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:30.997650 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.021327 (+ 23677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.021333 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.021827 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.023705 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.023706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.023706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.023891 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.024081 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.024081 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.024082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.024128 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.025252 (+ 1124us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.025253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.025289 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.025355 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.025464 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260620 00:12:31.026247 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:45695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 45695
}
attrs {
promote: true
}
}
dest_uuid: "6ace22bf7b20425e973c73cde979f0f7"
cas_config_opid_index: 5
I20260620 00:12:31.026490 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:31.026857 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f9a5161bdfd4b35a4b85c50546a0a03 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.027305 7178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:31.027864 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:45695
I20260620 00:12:31.027938 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.028004 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:31.028236 7136 master.cc:537] Master@127.0.98.60:45695 shutting down...
I20260620 00:12:31.028270 7178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 1 FOLLOWER]: Refusing update from remote peer 6ace22bf7b20425e973c73cde979f0f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 00:12:31.028529 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:12:31.029233 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 -> Peer 3f9a5161bdfd4b35a4b85c50546a0a03 (127.0.98.60:45695): Couldn't send request to peer 3f9a5161bdfd4b35a4b85c50546a0a03. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:12:31.032446 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.026926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.0.1:52172)
0620 00:12:31.027106 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.027116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.027150 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.028647 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.028648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.028650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.028694 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.028695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.029684 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.029685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.031020 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.031024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.032108 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.032109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.032188 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.032214 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.032282 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.032338 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 00:12:31.033119 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.027023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.0.98.61:43293)
0620 00:12:31.028434 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.028436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.028446 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:31.028472 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.028473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.028496 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.028585 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.028788 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.029572 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.031199 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.031909 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.031987 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.032071 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.032965 (+ 894us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.033023 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1382}
I20260620 00:12:31.033435 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 LEARNER]: Refusing update from remote peer 6ace22bf7b20425e973c73cde979f0f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 00:12:31.033699 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 00:12:31.033991 7136 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:31.034096 7136 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:31.034147 7136 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: stopping tablet replica
I20260620 00:12:31.034322 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3f9a5161bdfd4b35a4b85c50546a0a03 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.035470 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ace22bf7b20425e973c73cde979f0f7. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.035552 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:12:31.035588 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: true } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.035624 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:12:31.035801 7044 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: attempting to promote NON_VOTER f8cc7382290d4f10bec0f8e0917a5979 to VOTER
I20260620 00:12:31.036002 7044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:31.036617 7092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 LEARNER]: Refusing update from remote peer 6ace22bf7b20425e973c73cde979f0f7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 00:12:31.037205 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 00:12:31.037290 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.037105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.0.1:39162)
0620 00:12:31.037148 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.037186 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 00:12:31.037359 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 -> Peer 3f9a5161bdfd4b35a4b85c50546a0a03 (127.0.98.60:45695): Couldn't send request to peer 3f9a5161bdfd4b35a4b85c50546a0a03. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45695: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 00:12:31.037961 7233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f8cc7382290d4f10bec0f8e0917a5979 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.038079 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f8cc7382290d4f10bec0f8e0917a5979 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.039193 7232 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.039337 7232 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:12:31.039371 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.039471 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:31.039609 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ace22bf7b20425e973c73cde979f0f7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.039678 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:12:31.048203 7136 master.cc:552] Master@127.0.98.60:45695 shutdown complete.
I20260620 00:12:31.048278 7136 master_runner.cc:305] Clearing existing system tablet
I20260620 00:12:31.048411 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:31.048457 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:31.049275 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:31.049840 7235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.050020 7136 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:31.050084 7136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:31.050113 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "c85ea7aa16b9b7047f09dc2ec8dfd580"
server_key_iv: "a60386010fe7aee2affa2dc0239eb03e"
server_key_version: "encryptionkey@0"
I20260620 00:12:31.050869 7136 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:31.052870 7136 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:12:31.052944 7136 log.cc:1196] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 00:12:31.053166 7136 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Deleting consensus metadata
I20260620 00:12:31.053292 7136 master_runner.cc:315] Copying system tablet from 127.0.98.62:33771
I20260620 00:12:31.053867 7136 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33771
I20260620 00:12:31.061766 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.054243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.0.98.62:33771)
0620 00:12:31.054292 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.054294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.054350 (+ 56us) server_negotiation.cc:407] Connection header received
0620 00:12:31.054416 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.054417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.054428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.054503 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.054875 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.055635 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.056845 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.057137 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.057225 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.058569 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.058572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.058572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.058580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.060425 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.060879 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.060881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.060881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.060949 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.061152 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.061153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.061153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.061300 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.061367 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.061592 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.061674 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:12:31.061780 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.054170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56158)
0620 00:12:31.054294 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.054304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.054362 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.054522 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.054523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.054718 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.054788 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.054791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.055735 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.055736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.056665 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.056668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.057232 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.057233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.057317 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.058019 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.058025 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.058459 (+ 434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.060525 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.060526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.060526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.060799 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.061022 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.061022 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.061023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.061087 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.061428 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.061429 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.061479 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.061551 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.061659 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 00:12:31.062146 6504 tablet_copy_service.cc:140] P 6ace22bf7b20425e973c73cde979f0f7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f9a5161bdfd4b35a4b85c50546a0a03 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56158)
I20260620 00:12:31.062206 6504 tablet_copy_service.cc:161] P 6ace22bf7b20425e973c73cde979f0f7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f9a5161bdfd4b35a4b85c50546a0a03 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56158: session id = 3f9a5161bdfd4b35a4b85c50546a0a03-00000000000000000000000000000000
I20260620 00:12:31.062772 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:12:31.063403 7136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:12:31.064890 7136 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: tablet copy: Starting download of 0 data blocks...
I20260620 00:12:31.065052 7136 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: tablet copy: Starting download of 1 WAL segments...
I20260620 00:12:31.065865 7136 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:12:31.066803 6504 tablet_copy_service.cc:342] P 6ace22bf7b20425e973c73cde979f0f7: Request end of tablet copy session 3f9a5161bdfd4b35a4b85c50546a0a03-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:56158
I20260620 00:12:31.066865 6504 tablet_copy_service.cc:433] P 6ace22bf7b20425e973c73cde979f0f7: ending tablet copy session 3f9a5161bdfd4b35a4b85c50546a0a03-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f9a5161bdfd4b35a4b85c50546a0a03
I20260620 00:12:31.069537 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:31.070814 7242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.070797 7245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.070933 7136 server_base.cc:1034] running on GCE node
W20260620 00:12:31.077823 7243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.078040 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:31.078209 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:31.079411 7136 hybrid_clock.cc:648] HybridClock initialized: now 1781914351079402 us; error 28 us; skew 500 ppm
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:31.081975 7136 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:12:31.082487 7136 webserver.cc:460] Webserver started at http://127.0.98.60:36257/ using document root <none> and password file <none>
I20260620 00:12:31.082762 7136 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:31.082901 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:31.084147 7136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:12:31.087484 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.087607 7136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 00:12:31.087638 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "3f9a5161bdfd4b35a4b85c50546a0a03"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "c85ea7aa16b9b7047f09dc2ec8dfd580"
server_key_iv: "a60386010fe7aee2affa2dc0239eb03e"
server_key_version: "encryptionkey@0"
I20260620 00:12:31.087692 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:31.106647 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:31.108672 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:31.111898 7136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45695
I20260620 00:12:31.112366 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:12:31.113734 7305 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:12:31.114267 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Bootstrap starting.
I20260620 00:12:31.116708 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45695 every 8 connection(s)
I20260620 00:12:31.117868 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:12:31.118136 7305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: Bootstrap complete.
I20260620 00:12:31.118433 7305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:31.118549 7305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f9a5161bdfd4b35a4b85c50546a0a03, State: Initialized, Role: LEARNER
I20260620 00:12:31.118597 7305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:31.118892 7305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:12:31.118896 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } } }
I20260620 00:12:31.119076 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:12:31.120793 7319 catalog_manager.cc:1259] Loaded cluster ID: 175ef38ffff9424f8354a31b4af60900
I20260620 00:12:31.120890 7319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: loading cluster ID for follower catalog manager: success
I20260620 00:12:31.122221 7319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: acquiring CA information for follower catalog manager: success
I20260620 00:12:31.122577 7319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f9a5161bdfd4b35a4b85c50546a0a03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 00:12:31.137791 7230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:31.138017 7230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:31.138043 7230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:31.139276 7230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:31.139328 7230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:31.139357 7230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:12:31.141803 7230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39493
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:31.142122 7230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:31.142489 7230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:31.149870 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.150128 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.149861 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.150350 7230 server_base.cc:1034] running on GCE node
I20260620 00:12:31.150802 7230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:31.151116 7230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:31.152302 7230 hybrid_clock.cc:648] HybridClock initialized: now 1781914351152280 us; error 34 us; skew 500 ppm
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:31.155416 7230 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:12:31.155944 7230 webserver.cc:460] Webserver started at http://127.0.98.1:39493/ using document root <none> and password file <none>
I20260620 00:12:31.156208 7230 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:31.156283 7230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:31.157972 7230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:31.158610 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.158826 7230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:31.158982 7230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "c2a44fa171554e048562cfed9e9ad5d6"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "867ef330eb79a0b91ddc59461cfaa3ef"
server_key_iv: "b6fe5231d29da581873c99b5c5947e9c"
server_key_version: "encryptionkey@0"
I20260620 00:12:31.159333 7230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:31.170050 7230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:31.172866 7230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:31.173024 7230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:31.173277 7230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:31.173609 7230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:31.173643 7230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.173687 7230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:31.173713 7230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.179751 7230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39081
I20260620 00:12:31.179838 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39081 every 8 connection(s)
I20260620 00:12:31.180217 7230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:12:31.183161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 7230
I20260620 00:12:31.183319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6680
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:31.189886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45601
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33771
--webserver_port=46213
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:12:31.193332 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38019 (local address 127.0.98.62:33771)
0620 00:12:31.181234 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.181236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.181563 (+ 327us) server_negotiation.cc:407] Connection header received
0620 00:12:31.182127 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.182128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.182140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.182172 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.183148 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.183954 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.185631 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.185849 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.185934 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.188579 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.188581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.188581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.188589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.190547 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.192246 (+ 1699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.192247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.192247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.192317 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.192631 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.192632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.192632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.192770 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.192850 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.193152 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.193237 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:12:31.193547 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:38019)
0620 00:12:31.181388 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.181435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.181994 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.182274 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.182279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.182816 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.183033 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.183040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.184083 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.184084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.185517 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.185519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.185945 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.185946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.186076 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.186688 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.186697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.188408 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.190972 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.190980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.190983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.192160 (+ 1177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.192401 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.192403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.192403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.192524 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.192853 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.192860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.193007 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.193131 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.193306 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":43,"thread_start_us":96,"threads_started":1}
I20260620 00:12:31.193984 7449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
I20260620 00:12:31.194089 7449 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.194300 7449 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:31.194796 6484 ts_manager.cc:194] Re-registered known tserver with Master: c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081)
I20260620 00:12:31.195319 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40375 (local address 127.0.98.61:43293)
0620 00:12:31.181321 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.181323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.181541 (+ 218us) server_negotiation.cc:407] Connection header received
0620 00:12:31.182123 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.182124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.182137 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.182171 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.185061 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.185877 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.186924 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.187208 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.187245 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.189545 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.189546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.189547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.189554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.191821 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.193941 (+ 2120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.193943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.193944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.194013 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.194629 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.194630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.194631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.194806 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.194878 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.195086 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.195180 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:12:31.195604 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38019
I20260620 00:12:31.195712 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.98.1:40375)
0620 00:12:31.181405 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.181434 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.181994 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.182279 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.182279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.182824 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.184943 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.184946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.185984 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.185985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.186825 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.186827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.187319 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.187320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.187387 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.187886 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.187891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.189456 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.191964 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.191965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.191965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.193824 (+ 1859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.194149 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.194150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.194150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.194213 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.194910 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.194911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.194962 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.195427 (+ 465us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.195555 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":75,"thread_start_us":113,"threads_started":1}
I20260620 00:12:31.196122 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43293
I20260620 00:12:31.196175 7450 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.196477 7450 heartbeater.cc:511] Master 127.0.98.61:43293 requested a full tablet report, sending...
I20260620 00:12:31.196926 7082 ts_manager.cc:194] Registered new tserver with Master: c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081)
I20260620 00:12:31.197894 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34927 (local address 127.0.98.60:45695)
0620 00:12:31.181320 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.181322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.181583 (+ 261us) server_negotiation.cc:407] Connection header received
0620 00:12:31.182103 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.182104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.182119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.182196 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.183139 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.190161 (+ 7022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.191196 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.191543 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.191616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.193998 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.194000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.194000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.194008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.196619 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.197054 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.197056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.197057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.197125 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.197374 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.197375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.197375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.197493 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.197551 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.197744 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.197825 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
I20260620 00:12:31.198181 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.98.1:34927)
0620 00:12:31.181441 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.181462 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.182015 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.182568 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.182569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.182814 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.183031 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.183035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.190264 (+ 7229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.190265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.191104 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.191106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.191674 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.191676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.191750 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.192145 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.192150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.193863 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.196705 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.196706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.196707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.196958 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.197202 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.197202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.197203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.197265 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.197573 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.197574 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.197636 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.197729 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.197864 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":201,"thread_start_us":62,"threads_started":1}
I20260620 00:12:31.198616 7448 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45695
I20260620 00:12:31.198701 7448 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.198789 7448 heartbeater.cc:511] Master 127.0.98.60:45695 requested a full tablet report, sending...
I20260620 00:12:31.199216 7269 ts_manager.cc:194] Registered new tserver with Master: c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081)
W20260620 00:12:31.278538 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:31.278764 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:31.278791 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:31.279939 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:31.279981 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:31.280009 7456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:12:31.281487 7456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46213
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:31.281699 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:31.281886 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:31.287389 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.287340 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.287348 7456 server_base.cc:1034] running on GCE node
W20260620 00:12:31.287340 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.287839 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:31.288154 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:31.289333 7456 hybrid_clock.cc:648] HybridClock initialized: now 1781914351289310 us; error 36 us; skew 500 ppm
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:31.291819 7456 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:12:31.292313 7456 webserver.cc:460] Webserver started at http://127.0.98.2:46213/ using document root <none> and password file <none>
I20260620 00:12:31.292608 7456 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:31.292704 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:31.294137 7456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:31.294931 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.295190 7456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:12:31.295298 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "5159394615d747c39936d0bdf6fa4737"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "528faa9874ff4ac34577825053495197"
server_key_iv: "1ca6a94351aeb604015b40da83af263f"
server_key_version: "encryptionkey@0"
I20260620 00:12:31.295698 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:31.312458 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:31.315186 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:31.315354 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:31.315629 7456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:31.315953 7456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:31.316010 7456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.316044 7456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:31.316072 7456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.322484 7456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45601
I20260620 00:12:31.322597 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45601 every 8 connection(s)
I20260620 00:12:31.322928 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:12:31.326776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 7456
I20260620 00:12:31.326941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6814
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:12:31.334151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45307
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33771
--webserver_port=37177
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:12:31.335117 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.324188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45309 (local address 127.0.98.62:33771)
0620 00:12:31.324257 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.324258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.324708 (+ 450us) server_negotiation.cc:407] Connection header received
0620 00:12:31.325123 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.325124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.325136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.325167 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.326080 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.326769 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328002 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.328279 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328343 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.331382 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.331383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.331383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.331391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.333613 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334035 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.334036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.334037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.334098 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334422 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.334423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.334423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.334576 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.334623 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.334934 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.335017 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:12:31.335206 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.323761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39667 (local address 127.0.98.60:45695)
0620 00:12:31.323800 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.323802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.324120 (+ 318us) server_negotiation.cc:407] Connection header received
0620 00:12:31.324906 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.324908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.324926 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.325091 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.326303 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.327008 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328146 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.328450 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328535 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.331322 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.331324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.331324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.331333 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.333193 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334106 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.334107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.334108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.334166 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334433 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.334435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.334435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.334592 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.334638 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.334942 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.335060 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:31.335355 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.324098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:45309)
0620 00:12:31.324631 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.324641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.325032 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.325226 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.325227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.325714 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.325966 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.325972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.326888 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.326889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.327902 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.327904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.328364 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.328517 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.329198 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.329208 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.331264 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.333698 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.333699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.333700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.333938 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.334185 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334185 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.334186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.334344 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.334662 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.334667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.334797 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.334903 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.335082 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":63,"threads_started":1}
I20260620 00:12:31.335556 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.323719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.98.2:39667)
0620 00:12:31.323954 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.323992 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.324726 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.325206 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.325211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.325721 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.325966 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.325972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.327103 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.327104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.328061 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.328064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.328528 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.328529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.328619 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.329251 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.329256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.331224 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.333312 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.333321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.333324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.333989 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.334240 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.334242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.334242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.334350 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.334686 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.334687 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.334805 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.335286 (+ 481us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.335406 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":16,"thread_start_us":133,"threads_started":1}
I20260620 00:12:31.336036 7585 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45695
I20260620 00:12:31.336061 7586 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
I20260620 00:12:31.336156 7585 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.336156 7586 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.336354 7585 heartbeater.cc:511] Master 127.0.98.60:45695 requested a full tablet report, sending...
I20260620 00:12:31.336354 7586 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:31.336830 6484 ts_manager.cc:194] Re-registered known tserver with Master: 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
I20260620 00:12:31.336911 7269 ts_manager.cc:194] Registered new tserver with Master: 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
I20260620 00:12:31.337307 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45309
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:12:31.342140 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.323719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.98.2:45415)
0620 00:12:31.324236 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.324247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.324735 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.325206 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.325211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.325713 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.326297 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.326300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.335776 (+ 9476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.335778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.336661 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.336663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.337199 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.337200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.337267 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.337768 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.337774 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.338975 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.341083 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.341084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.341084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.341262 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.341486 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.341487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.341487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.341539 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.341841 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.341842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.341880 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.341940 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.342040 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":19,"spinlock_wait_cycles":7936,"thread_start_us":128,"threads_started":1}
I20260620 00:12:31.342207 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.323711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45415 (local address 127.0.98.61:43293)
0620 00:12:31.323772 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.323773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.324305 (+ 532us) server_negotiation.cc:407] Connection header received
0620 00:12:31.324997 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.324999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.325013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.325074 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.326398 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.327084 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.336740 (+ 9656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.337007 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.337093 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.339074 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.339076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.339076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.339085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.340968 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.341345 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.341347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.341347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.341415 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.341602 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.341603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.341604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.341768 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.341841 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.341989 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.342074 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:12:31.342557 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43293
I20260620 00:12:31.342624 7587 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.342730 7587 heartbeater.cc:511] Master 127.0.98.61:43293 requested a full tablet report, sending...
I20260620 00:12:31.343083 7082 ts_manager.cc:194] Registered new tserver with Master: 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
W20260620 00:12:31.420850 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:12:31.421124 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:12:31.421165 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:12:31.422413 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:12:31.422490 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:12:31.422539 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:12:31.424168 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37177
--tserver_master_addrs=127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:12:31.424449 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:12:31.424752 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:12:31.430543 7597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.430557 7598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:12:31.430575 7600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:12:31.430627 7592 server_base.cc:1034] running on GCE node
I20260620 00:12:31.431107 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:12:31.431373 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:12:31.432590 7592 hybrid_clock.cc:648] HybridClock initialized: now 1781914351432522 us; error 65 us; skew 500 ppm
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:12:31.435371 7592 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:12:31.435869 7592 webserver.cc:460] Webserver started at http://127.0.98.3:37177/ using document root <none> and password file <none>
I20260620 00:12:31.436123 7592 fs_manager.cc:359] Metadata directory not provided
I20260620 00:12:31.436216 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:12:31.437781 7592 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:31.438526 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.438781 7592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:12:31.438920 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "aaf420f438df4751b5d23ce4f5d398e4"
format_stamp: "Formatted at 2026-06-20 00:12:30 on dist-test-slave-28bv"
server_key: "e2972f03ce085296edea02da70f47518"
server_key_iv: "b2318865a7826a12ec59f9a48aface78"
server_key_version: "encryptionkey@0"
I20260620 00:12:31.439301 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:12:31.451665 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:12:31.454280 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:12:31.454463 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:12:31.454706 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:12:31.455027 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:12:31.455082 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.455127 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:12:31.455169 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:12:31.461570 7592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45307
I20260620 00:12:31.461643 7720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45307 every 8 connection(s)
I20260620 00:12:31.462080 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:12:31 dist-test-slave-28bv krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914351, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:12:31.471549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 7592
I20260620 00:12:31.473490 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.463410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58633 (local address 127.0.98.61:43293)
0620 00:12:31.463465 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.463467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.463476 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:31.463907 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.463908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.463920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.463979 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.464776 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465549 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.466668 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.466967 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.467027 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.470188 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.470190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.470190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.470199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.471891 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472418 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.472419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.472420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.472479 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472732 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.472733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.472733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.472861 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.472911 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.473303 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.473394 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:12:31.473526 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.463525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35085 (local address 127.0.98.60:45695)
0620 00:12:31.463576 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.463578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.463588 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:31.463950 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.463952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.463965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.463999 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.464788 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465309 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.466694 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.467001 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.467064 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.469844 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.469846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.469847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.469856 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.471699 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472240 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.472242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.472243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.472311 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472757 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.472758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.472758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.472906 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.473063 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.473305 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.473395 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:12:31.474130 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.463238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45695 (local address 127.0.98.3:35085)
0620 00:12:31.463351 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.463361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.463829 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.464145 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.464147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.464416 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.464676 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.464683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465411 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.465412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.466579 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.466580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.467120 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.467121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.467189 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.467950 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.467954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.469692 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.471801 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.471811 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.471814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.472139 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.472390 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472390 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.472391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.472537 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.473057 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.473058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.473129 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.473636 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.473852 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":51,"thread_start_us":69,"threads_started":1}
I20260620 00:12:31.474635 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.462884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41599 (local address 127.0.98.62:33771)
0620 00:12:31.462945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.462948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.463227 (+ 279us) server_negotiation.cc:407] Connection header received
0620 00:12:31.463950 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.463952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.463965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.463999 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.464828 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465587 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.466520 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.466814 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.466875 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.470861 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.470863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.470863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.470872 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.472671 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.473564 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.473641 (+ 77us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.473642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.473734 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.474014 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.474015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.474016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.474146 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.474193 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.474413 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.474506 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:12:31.474720 7721 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45695
I20260620 00:12:31.474890 7721 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.475095 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.462897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43293 (local address 127.0.98.3:58633)
0620 00:12:31.463075 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.463120 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.463786 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.464059 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.464065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.464369 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.464674 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.464679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465642 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.465643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.466577 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.466579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.467079 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.467079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.467133 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.467873 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.467877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.470109 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.472102 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.472104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.472320 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.472592 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.472593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.472593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.472656 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.472950 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.472956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.473127 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.474072 (+ 945us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.474961 (+ 889us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":34,"thread_start_us":193,"threads_started":1}
I20260620 00:12:31.475474 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43293
I20260620 00:12:31.475550 7723 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.475626 7723 heartbeater.cc:511] Master 127.0.98.61:43293 requested a full tablet report, sending...
I20260620 00:12:31.475148 7721 heartbeater.cc:511] Master 127.0.98.60:45695 requested a full tablet report, sending...
I20260620 00:12:31.476133 7082 ts_manager.cc:194] Registered new tserver with Master: aaf420f438df4751b5d23ce4f5d398e4 (127.0.98.3:45307)
I20260620 00:12:31.476280 7269 ts_manager.cc:194] Registered new tserver with Master: aaf420f438df4751b5d23ce4f5d398e4 (127.0.98.3:45307)
I20260620 00:12:31.476853 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.462897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:41599)
0620 00:12:31.463077 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.463156 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.463829 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.464486 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.464487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.464672 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:12:31.464727 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.464729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.465676 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.465677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.466396 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.466398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.466942 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.466944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.467112 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.467854 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:12:31.467865 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:12:31.470740 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:12:31.473091 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.473092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.473092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.473445 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.473870 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:12:31.473870 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:12:31.473871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:12:31.473931 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:12:31.474244 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:12:31.474245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:12:31.474287 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.474374 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.476186 (+ 1812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1705,"thread_start_us":102,"threads_started":1}
I20260620 00:12:31.477386 7722 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33771
I20260620 00:12:31.477447 7722 heartbeater.cc:464] Registering TS with master...
I20260620 00:12:31.477525 7722 heartbeater.cc:511] Master 127.0.98.62:33771 requested a full tablet report, sending...
I20260620 00:12:31.477772 6484 ts_manager.cc:194] Re-registered known tserver with Master: aaf420f438df4751b5d23ce4f5d398e4 (127.0.98.3:45307)
I20260620 00:12:31.478281 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41599
I20260620 00:12:31.484489 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.475894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.0.98.62:33771)
0620 00:12:31.475930 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.475932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.476610 (+ 678us) server_negotiation.cc:407] Connection header received
0620 00:12:31.476694 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.476695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.476706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.476796 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.478021 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.478703 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.479543 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.479758 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.479841 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.481463 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.481464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.481465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.481473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.483148 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.483519 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.483520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.483520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.483584 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.483810 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.483811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.483812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.483951 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.484050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.484279 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.484359 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:12:31.485097 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.475681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.0.98.60:45695)
0620 00:12:31.475717 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.475719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.477219 (+ 1500us) server_negotiation.cc:407] Connection header received
0620 00:12:31.477303 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.477304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.477315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.477375 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.478172 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.478926 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.480076 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.480283 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.480361 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.481822 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.481823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.481824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.481831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.483820 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.484208 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.484210 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.484211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.484270 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.484483 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.484484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.484485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.484656 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.484745 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.484914 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.484985 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:12:31.486789 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.477089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.98.61:43293)
0620 00:12:31.477134 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.477136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.477144 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:12:31.477163 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.477164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.477175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.477233 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.478172 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.478934 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.480057 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.480986 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.481068 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.482371 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.482373 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.482374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.482382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.484103 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.484670 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.484671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.484672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.484730 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.484921 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.484922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.484922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.485056 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.485129 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.486608 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.486688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:12:31.486392 6484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:12:31.488526 6484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:12:31.496774 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45307 (local address 127.0.0.1:32816)
0620 00:12:31.492347 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.492362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.492403 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.493383 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.493384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.493385 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.493420 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.493421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494495 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.494496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.495764 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.495767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.496511 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.496512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.496588 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.496593 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.496626 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.496679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":59}
I20260620 00:12:31.496827 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.0.98.2:45601)
0620 00:12:31.493522 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.493527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.493536 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:31.493571 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.493573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.493588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.493664 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.493788 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494726 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.495641 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.496386 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.496468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.496571 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.496702 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.496741 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":85,"threads_started":1}
I20260620 00:12:31.496774 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45601 (local address 127.0.0.1:42372)
0620 00:12:31.492759 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.492769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.492804 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.493675 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.493676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.493678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.493724 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.493726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494818 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.494819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.495547 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.495549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.496481 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.496483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.496613 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.496618 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.496672 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.496702 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260620 00:12:31.497311 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.0.98.3:45307)
0620 00:12:31.493215 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.493220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.493229 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:12:31.493268 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.493270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.493286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.493364 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.493503 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494378 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.495925 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.496396 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.497004 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.497118 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.497145 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.497190 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":127,"threads_started":1}
I20260620 00:12:31.497855 7655 tablet_service.cc:1467] Processing CreateTablet for tablet 7df436f2f3f3427e8536dee3ee658efe (DEFAULT_TABLE table=test-table [id=5a444e143c494745a3e485abb5ab8fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:31.497965 7519 tablet_service.cc:1467] Processing CreateTablet for tablet 7df436f2f3f3427e8536dee3ee658efe (DEFAULT_TABLE table=test-table [id=5a444e143c494745a3e485abb5ab8fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:31.498262 7655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df436f2f3f3427e8536dee3ee658efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:31.498327 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df436f2f3f3427e8536dee3ee658efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:31.501397 7744 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Bootstrap starting.
I20260620 00:12:31.501662 7745 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Bootstrap starting.
I20260620 00:12:31.501885 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39081 (local address 127.0.0.1:56172)
0620 00:12:31.492626 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.492636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.492684 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.493088 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.493089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.493091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.493149 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.493150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494450 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.494451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.495766 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.495767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.497364 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.497365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.501677 (+ 4312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.501684 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.501747 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.501803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":70,"thread_start_us":101,"threads_started":1}
I20260620 00:12:31.501902 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.492676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.0.98.1:39081)
0620 00:12:31.492890 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.492911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.492921 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:31.492967 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.492970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.492987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.493080 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.493214 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.494357 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.495853 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.497261 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.497362 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.497438 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.501771 (+ 4333us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.501826 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20260620 00:12:31.502548 7382 tablet_service.cc:1467] Processing CreateTablet for tablet 7df436f2f3f3427e8536dee3ee658efe (DEFAULT_TABLE table=test-table [id=5a444e143c494745a3e485abb5ab8fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:12:31.502866 7382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7df436f2f3f3427e8536dee3ee658efe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:12:31.502867 7744 tablet_bootstrap.cc:654] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:31.503159 7744 log.cc:824] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:31.503924 7745 tablet_bootstrap.cc:654] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:31.504004 7744 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: No bootstrap required, opened a new log
I20260620 00:12:31.504074 7744 ts_tablet_manager.cc:1397] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:12:31.504240 7745 log.cc:824] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:31.505065 7745 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: No bootstrap required, opened a new log
I20260620 00:12:31.505142 7745 ts_tablet_manager.cc:1397] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 00:12:31.505987 7748 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Bootstrap starting.
I20260620 00:12:31.506264 7744 raft_consensus.cc:348] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.506444 7744 raft_consensus.cc:374] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:31.506489 7744 raft_consensus.cc:729] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf420f438df4751b5d23ce4f5d398e4, State: Initialized, Role: FOLLOWER
I20260620 00:12:31.506628 7744 consensus_queue.cc:260] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.506780 7748 tablet_bootstrap.cc:654] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Neither blocks nor log segments found. Creating new log.
I20260620 00:12:31.506896 7744 ts_tablet_manager.cc:1428] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:31.507026 7722 heartbeater.cc:503] Master 127.0.98.62:33771 was elected leader, sending a full tablet report...
I20260620 00:12:31.507148 7745 raft_consensus.cc:348] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.507313 7745 raft_consensus.cc:374] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:31.507359 7745 raft_consensus.cc:729] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5159394615d747c39936d0bdf6fa4737, State: Initialized, Role: FOLLOWER
I20260620 00:12:31.507539 7745 consensus_queue.cc:260] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.507983 7745 ts_tablet_manager.cc:1428] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:12:31.508483 7586 heartbeater.cc:503] Master 127.0.98.62:33771 was elected leader, sending a full tablet report...
I20260620 00:12:31.509114 7748 log.cc:824] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:12:31.509915 7748 tablet_bootstrap.cc:492] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: No bootstrap required, opened a new log
I20260620 00:12:31.510011 7748 ts_tablet_manager.cc:1397] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 00:12:31.512679 7748 raft_consensus.cc:348] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.512856 7748 raft_consensus.cc:374] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:12:31.512914 7748 raft_consensus.cc:729] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a44fa171554e048562cfed9e9ad5d6, State: Initialized, Role: FOLLOWER
I20260620 00:12:31.512935 7749 raft_consensus.cc:482] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:12:31.513056 7748 consensus_queue.cc:260] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.513118 7749 raft_consensus.cc:504] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.513350 7748 ts_tablet_manager.cc:1428] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:12:31.513657 7449 heartbeater.cc:503] Master 127.0.98.62:33771 was elected leader, sending a full tablet report...
I20260620 00:12:31.514304 7749 leader_election.cc:288] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081), 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
I20260620 00:12:31.517311 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.514217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46997 (local address 127.0.98.1:39081)
0620 00:12:31.514256 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.514258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.514269 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:12:31.514293 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.514294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.514303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.514350 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.514652 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.515350 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.516605 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.516961 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.517018 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.517026 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.517201 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.517246 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:31.517321 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.513881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39081 (local address 127.0.98.3:46997)
0620 00:12:31.514202 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.514212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.514248 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.514498 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.514499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.514501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.514559 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.514561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.515436 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.515437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.516478 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.516480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.517045 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.517046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.517129 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.517132 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.517190 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.517226 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":276}
I20260620 00:12:31.517402 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.514128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43505 (local address 127.0.98.2:45601)
0620 00:12:31.514198 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.514201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.514211 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:12:31.514238 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.514239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.514252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.514308 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:12:31.514467 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.515104 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.516578 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.516988 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.517090 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.517101 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.517289 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.517332 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:12:31.517439 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.513881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45601 (local address 127.0.98.3:43505)
0620 00:12:31.513989 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:31.514002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:12:31.514045 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:12:31.514313 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:12:31.514315 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:12:31.514318 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:12:31.514407 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.514410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.515168 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.515169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.516468 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:12:31.516470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.517117 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.517117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:12:31.517210 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.517213 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:12:31.517269 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:12:31.517310 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":26}
I20260620 00:12:31.517696 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df436f2f3f3427e8536dee3ee658efe" candidate_uuid: "aaf420f438df4751b5d23ce4f5d398e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a44fa171554e048562cfed9e9ad5d6" is_pre_election: true
I20260620 00:12:31.517836 7402 raft_consensus.cc:2454] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf420f438df4751b5d23ce4f5d398e4 in term 0.
I20260620 00:12:31.518069 7610 leader_election.cc:302] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaf420f438df4751b5d23ce4f5d398e4, c2a44fa171554e048562cfed9e9ad5d6; no voters:
I20260620 00:12:31.518091 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df436f2f3f3427e8536dee3ee658efe" candidate_uuid: "aaf420f438df4751b5d23ce4f5d398e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5159394615d747c39936d0bdf6fa4737" is_pre_election: true
I20260620 00:12:31.518209 7749 raft_consensus.cc:2781] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:12:31.518239 7539 raft_consensus.cc:2454] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf420f438df4751b5d23ce4f5d398e4 in term 0.
I20260620 00:12:31.518265 7749 raft_consensus.cc:482] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:12:31.518291 7749 raft_consensus.cc:3037] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:31.519156 7749 raft_consensus.cc:504] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.519304 7749 leader_election.cc:288] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [CANDIDATE]: Term 1 election: Requested vote from peers c2a44fa171554e048562cfed9e9ad5d6 (127.0.98.1:39081), 5159394615d747c39936d0bdf6fa4737 (127.0.98.2:45601)
I20260620 00:12:31.519430 7402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df436f2f3f3427e8536dee3ee658efe" candidate_uuid: "aaf420f438df4751b5d23ce4f5d398e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a44fa171554e048562cfed9e9ad5d6"
I20260620 00:12:31.519441 7539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df436f2f3f3427e8536dee3ee658efe" candidate_uuid: "aaf420f438df4751b5d23ce4f5d398e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5159394615d747c39936d0bdf6fa4737"
I20260620 00:12:31.519539 7402 raft_consensus.cc:3037] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:31.519552 7539 raft_consensus.cc:3037] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:12:31.520246 7402 raft_consensus.cc:2454] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf420f438df4751b5d23ce4f5d398e4 in term 1.
I20260620 00:12:31.520417 7610 leader_election.cc:302] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaf420f438df4751b5d23ce4f5d398e4, c2a44fa171554e048562cfed9e9ad5d6; no voters:
I20260620 00:12:31.520521 7749 raft_consensus.cc:2781] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:12:31.520687 7749 raft_consensus.cc:686] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 1 LEADER]: Becoming Leader. State: Replica: aaf420f438df4751b5d23ce4f5d398e4, State: Running, Role: LEADER
I20260620 00:12:31.520792 7749 consensus_queue.cc:237] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } }
I20260620 00:12:31.520875 7539 raft_consensus.cc:2454] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf420f438df4751b5d23ce4f5d398e4 in term 1.
I20260620 00:12:31.521672 6484 catalog_manager.cc:5495] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaf420f438df4751b5d23ce4f5d398e4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aaf420f438df4751b5d23ce4f5d398e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf420f438df4751b5d23ce4f5d398e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 } health_report { overall_health: UNKNOWN } } }
I20260620 00:12:31.535343 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.532985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.0.98.3:45307)
0620 00:12:31.533041 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.533043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.533072 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:12:31.533141 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.533142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.533161 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.533218 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:12:31.533391 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.533860 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.534609 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.534786 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.534835 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.534968 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:12:31.535065 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:12:31.535135 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.535211 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.535229 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:12:31.537411 7402 raft_consensus.cc:1261] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 1 FOLLOWER]: Refusing update from remote peer aaf420f438df4751b5d23ce4f5d398e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:31.537451 7539 raft_consensus.cc:1261] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 1 FOLLOWER]: Refusing update from remote peer aaf420f438df4751b5d23ce4f5d398e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:12:31.537797 7749 consensus_queue.cc:1035] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a44fa171554e048562cfed9e9ad5d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:31.537933 7753 consensus_queue.cc:1035] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5159394615d747c39936d0bdf6fa4737" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:12:31.541049 7756 mvcc.cc:205] Tried to move back new op lower bound from 7298721183893987328 to 7298721183829749760. Current Snapshot: MvccSnapshot[applied={T|T < 7298721183893987328}]
I20260620 00:12:31.545388 6483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:31.545492 6483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:12:31.546903 6483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ace22bf7b20425e973c73cde979f0f7: Sending DeleteTablet for 3 replicas of tablet 7df436f2f3f3427e8536dee3ee658efe
I20260620 00:12:31.547252 7382 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df436f2f3f3427e8536dee3ee658efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56172
I20260620 00:12:31.547289 7519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df436f2f3f3427e8536dee3ee658efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42372
I20260620 00:12:31.547571 7655 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df436f2f3f3427e8536dee3ee658efe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:12:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32816
I20260620 00:12:31.547689 7765 tablet_replica.cc:330] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: stopping tablet replica
I20260620 00:12:31.547801 7765 raft_consensus.cc:2229] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:31.547915 7765 raft_consensus.cc:2258] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:31.548389 7765 ts_tablet_manager.cc:1905] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:31.548456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6452
I20260620 00:12:31.551865 7765 ts_tablet_manager.cc:1918] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:31.551951 7765 log.cc:1196] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/7df436f2f3f3427e8536dee3ee658efe
I20260620 00:12:31.552227 7765 ts_tablet_manager.cc:1939] T 7df436f2f3f3427e8536dee3ee658efe P c2a44fa171554e048562cfed9e9ad5d6: Deleting consensus metadata
W20260620 00:12:31.555428 7335 connection.cc:535] server connection from 127.0.0.1:56172 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:12:31.556484 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.547738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.0.98.60:45695)
0620 00:12:31.547782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.547784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.547874 (+ 90us) server_negotiation.cc:407] Connection header received
0620 00:12:31.547975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.547976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.547987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.548015 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.548875 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.549598 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.550733 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.551030 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.551073 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.553394 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.553396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.553396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.553406 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.555220 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.555605 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.555606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.555606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.555674 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.555896 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.555897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.555897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.556071 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.556165 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.556288 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.556380 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:12:31.563508 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.556644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.0.98.61:43293)
0620 00:12:31.556698 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.556700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.556802 (+ 102us) server_negotiation.cc:407] Connection header received
0620 00:12:31.556858 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.556859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.556869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.556924 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.557692 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.558173 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.558894 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.559105 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.559182 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.560407 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.560408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.560408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.560415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.562011 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.562396 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.562397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.562397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.562456 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.562800 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.562801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.562801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.562947 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.563045 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.563228 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.563403 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:12:31.564685 7767 tablet_replica.cc:330] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: stopping tablet replica
I20260620 00:12:31.564858 7766 tablet_replica.cc:330] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: stopping tablet replica
I20260620 00:12:31.565016 7766 raft_consensus.cc:2229] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:12:31.565092 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:31.557071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.98.60:45695)
0620 00:12:31.557106 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:12:31.557108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:12:31.557406 (+ 298us) server_negotiation.cc:407] Connection header received
0620 00:12:31.557498 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:12:31.557499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:12:31.557509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:12:31.557537 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:12:31.558270 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.558997 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.560381 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:12:31.560719 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:12:31.560759 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:12:31.562151 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:12:31.562153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:12:31.562153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:12:31.562160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:12:31.563837 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.564230 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.564231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.564232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.564290 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:12:31.564488 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:12:31.564488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:12:31.564489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:12:31.564635 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:12:31.564742 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 00:12:31.564914 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 00:12:31.564987 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:12:31.565241 7767 raft_consensus.cc:2229] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:12:31.565421 7766 raft_consensus.cc:2258] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:31.565578 7767 raft_consensus.cc:2258] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:12:31.566138 7767 ts_tablet_manager.cc:1905] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:31.566138 7766 ts_tablet_manager.cc:1905] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:12:31.567922 7767 ts_tablet_manager.cc:1918] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:31.567952 7766 ts_tablet_manager.cc:1918] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:12:31.568123 7767 log.cc:1196] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/7df436f2f3f3427e8536dee3ee658efe
I20260620 00:12:31.568128 7766 log.cc:1196] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/7df436f2f3f3427e8536dee3ee658efe
I20260620 00:12:31.568387 7767 ts_tablet_manager.cc:1939] T 7df436f2f3f3427e8536dee3ee658efe P aaf420f438df4751b5d23ce4f5d398e4: Deleting consensus metadata
I20260620 00:12:31.568404 7766 ts_tablet_manager.cc:1939] T 7df436f2f3f3427e8536dee3ee658efe P 5159394615d747c39936d0bdf6fa4737: Deleting consensus metadata
W20260620 00:12:31.568830 7611 connection.cc:441] server connection from 127.0.0.1:32816 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32816 (request call id 1) could send its response
W20260620 00:12:31.568840 7472 connection.cc:441] server connection from 127.0.0.1:42372 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42372 (request call id 1) could send its response
I20260620 00:12:32.543350 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:32.542893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:57473)
0620 00:12:32.543200 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:32.543288 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":67,"threads_started":1}
I20260620 00:12:32.543478 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:32.543197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:60445)
0620 00:12:32.543326 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:32.543428 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
W20260620 00:12:32.543578 7586 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
W20260620 00:12:32.543902 7449 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:32.543941 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:32.543614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:47711)
0620 00:12:32.543825 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:32.543896 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":68,"threads_started":1}
W20260620 00:12:32.544137 7722 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:33.047923 7779 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:33.048074 7779 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:33.048314 7779 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:33.048810 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:33.048568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56676)
0620 00:12:33.048685 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:33.048746 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
W20260620 00:12:33.048960 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:33.049394 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:33.049286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56690)
0620 00:12:33.049343 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:33.049373 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:12:33.049546 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:33.049703 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:33.049829 7779 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:33.544363 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:33.544005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:38663)
0620 00:12:33.544239 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:33.544304 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260620 00:12:33.544657 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:33.544202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:33701)
0620 00:12:33.544467 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:33.544598 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
I20260620 00:12:33.544874 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:33.544668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:50591)
0620 00:12:33.544777 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:33.544811 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260620 00:12:34.545153 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:34.544856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:33737)
0620 00:12:34.545054 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:34.545107 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260620 00:12:34.545377 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:34.545087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:58817)
0620 00:12:34.545285 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:34.545321 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260620 00:12:34.545553 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:34.545324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:57997)
0620 00:12:34.545445 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:34.545504 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260620 00:12:34.602909 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:34.603139 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:34.603333 7788 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:34.603757 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:34.603555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56900)
0620 00:12:34.603670 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:34.603719 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260620 00:12:34.604251 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:34.604108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:56902)
0620 00:12:34.604186 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:34.604218 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 00:12:34.604359 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:34.604439 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:34.604575 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:35.545910 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:35.545582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:46915)
0620 00:12:35.545793 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:35.545843 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":66,"threads_started":1}
I20260620 00:12:35.546041 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:35.545835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:36901)
0620 00:12:35.545955 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:35.545990 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260620 00:12:35.546106 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:35.545877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:35177)
0620 00:12:35.545987 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:35.546063 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260620 00:12:36.546573 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:36.546334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:50745)
0620 00:12:36.546457 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:36.546502 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260620 00:12:36.546734 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:36.546521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:36511)
0620 00:12:36.546621 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:36.546691 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260620 00:12:36.546768 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:36.546455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:39857)
0620 00:12:36.546590 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:36.546632 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":50,"threads_started":1}
I20260620 00:12:37.547349 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:37.547116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:48295)
0620 00:12:37.547257 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:37.547305 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
I20260620 00:12:37.547497 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:37.547253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:58925)
0620 00:12:37.547428 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:37.547457 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":48,"threads_started":1}
I20260620 00:12:37.547497 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:37.547247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:49817)
0620 00:12:37.547414 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:37.547452 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260620 00:12:37.785914 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:37.786020 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:37.786222 7800 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:37.786629 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:37.786435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:57174)
0620 00:12:37.786554 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:37.786593 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260620 00:12:37.787281 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:37.787150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:57196)
0620 00:12:37.787193 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:37.787248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 00:12:37.787364 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:37.787400 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:37.787484 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:38.548135 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:38.547871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:38215)
0620 00:12:38.548022 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:38.548066 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260620 00:12:38.548151 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:38.547871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:40769)
0620 00:12:38.548051 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:38.548094 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260620 00:12:38.548273 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:38.547871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:44055)
0620 00:12:38.548184 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:38.548224 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":96,"threads_started":1}
I20260620 00:12:39.548892 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:39.548638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:33003)
0620 00:12:39.548781 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:39.548850 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260620 00:12:39.548966 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:39.548717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:42381)
0620 00:12:39.548862 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:39.548907 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260620 00:12:39.548892 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:39.548638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:44791)
0620 00:12:39.548781 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:39.548828 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260620 00:12:40.549652 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:40.549423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:50289)
0620 00:12:40.549550 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:40.549591 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260620 00:12:40.549690 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:40.549418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:45823)
0620 00:12:40.549594 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:40.549637 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260620 00:12:40.549871 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:40.549656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:48989)
0620 00:12:40.549802 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:40.549832 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260620 00:12:41.550446 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:41.550189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:54915)
0620 00:12:41.550311 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:41.550349 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260620 00:12:41.550480 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:41.550220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:43831)
0620 00:12:41.550344 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:41.550417 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260620 00:12:41.550513 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:41.550189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:33833)
0620 00:12:41.550365 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:41.550468 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":64,"threads_started":1}
I20260620 00:12:42.551222 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:42.550972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:34331)
0620 00:12:42.551125 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:42.551167 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260620 00:12:42.551270 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:42.550990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:60143)
0620 00:12:42.551125 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:42.551174 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260620 00:12:42.551230 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:42.551011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:48101)
0620 00:12:42.551123 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:42.551178 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":49,"threads_started":1}
I20260620 00:12:42.788187 7818 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:42.788357 7818 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:42.788589 7818 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:42.788995 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:42.788730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:58734)
0620 00:12:42.788875 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:42.788956 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260620 00:12:42.789088 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260620 00:12:42.789562 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:42.789414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:58748)
0620 00:12:42.789506 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:42.789539 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 00:12:42.789651 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:42.789713 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:42.789806 7818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:43.551918 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:43.551662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:47085)
0620 00:12:43.551810 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:43.551854 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260620 00:12:43.552021 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:43.551697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:58903)
0620 00:12:43.551895 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:43.551970 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":51,"threads_started":1}
I20260620 00:12:43.552034 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:43.551840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:38609)
0620 00:12:43.551951 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:43.551989 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260620 00:12:44.552726 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:44.552443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:60931)
0620 00:12:44.552619 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:44.552665 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260620 00:12:44.552806 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:44.552443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:47147)
0620 00:12:44.552732 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:44.552765 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":71,"threads_started":1}
I20260620 00:12:44.552784 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:44.552587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:59147)
0620 00:12:44.552695 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:44.552733 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260620 00:12:45.553543 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:45.553293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:55957)
0620 00:12:45.553460 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:45.553497 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260620 00:12:45.553562 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:45.553221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:54219)
0620 00:12:45.553455 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:45.553515 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260620 00:12:45.553562 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:45.553400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:42189)
0620 00:12:45.553497 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:45.553525 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260620 00:12:46.554288 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:46.554052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:60077)
0620 00:12:46.554202 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:46.554237 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 00:12:46.554288 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:46.554052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:35137)
0620 00:12:46.554194 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:46.554237 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 00:12:46.554311 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:46.554051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:50621)
0620 00:12:46.554194 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:46.554238 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260620 00:12:47.555025 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:47.554718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:41083)
0620 00:12:47.554921 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:47.554964 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":64,"threads_started":1}
I20260620 00:12:47.555038 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:47.554699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:56777)
0620 00:12:47.554937 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:47.554980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
I20260620 00:12:47.555119 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:47.554727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:38765)
0620 00:12:47.554973 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:47.555077 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260620 00:12:48.555687 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:48.555399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:47763)
0620 00:12:48.555578 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:48.555625 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":125,"threads_started":1}
I20260620 00:12:48.555785 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:48.555492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:49259)
0620 00:12:48.555587 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:48.555631 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260620 00:12:48.555948 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:48.555610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:41903)
0620 00:12:48.555821 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:48.555863 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":62,"threads_started":1}
I20260620 00:12:49.415086 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:49.415227 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:49.415436 7839 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:49.415805 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:49.415585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:40980)
0620 00:12:49.415725 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:49.415769 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":47,"threads_started":1}
W20260620 00:12:49.415913 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:49.416491 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:49.416174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:40990)
0620 00:12:49.416431 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:49.416464 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
W20260620 00:12:49.416618 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:49.416698 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:49.416790 7839 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:49.556627 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:49.556186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:56953)
0620 00:12:49.556499 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:49.556566 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":125,"threads_started":1}
I20260620 00:12:49.556770 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:49.556540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:55725)
0620 00:12:49.556691 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:49.556727 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260620 00:12:49.556787 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:49.556503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:42527)
0620 00:12:49.556696 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:49.556739 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260620 00:12:50.557914 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:50.557677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:53211)
0620 00:12:50.557792 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:50.557851 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260620 00:12:50.558475 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:50.558067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:39163)
0620 00:12:50.558395 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:50.558429 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":44,"threads_started":1}
I20260620 00:12:50.558949 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:50.557624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:54767)
0620 00:12:50.558867 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:50.558905 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":67,"threads_started":1}
I20260620 00:12:51.558878 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:51.558518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:59005)
0620 00:12:51.558751 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:51.558796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":66,"threads_started":1}
I20260620 00:12:51.559196 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:51.558914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:40995)
0620 00:12:51.559104 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:51.559147 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260620 00:12:51.559897 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:51.559737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:54747)
0620 00:12:51.559838 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:51.559862 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260620 00:12:52.560017 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:52.559713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:50071)
0620 00:12:52.559916 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:52.559958 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
I20260620 00:12:52.560387 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:52.559793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:38421)
0620 00:12:52.560297 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:52.560338 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":125,"threads_started":1}
I20260620 00:12:52.560779 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:52.560365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:49425)
0620 00:12:52.560700 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:52.560738 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":50,"threads_started":1}
I20260620 00:12:53.561138 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:53.560909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:46911)
0620 00:12:53.561045 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:53.561080 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260620 00:12:53.561189 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:53.560860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:48395)
0620 00:12:53.560980 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:53.561145 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260620 00:12:53.561512 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:53.561266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:48211)
0620 00:12:53.561431 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:53.561468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":45,"threads_started":1}
I20260620 00:12:53.881002 7857 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:53.881116 7857 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:53.881304 7857 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:53.881736 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:53.881526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:41192)
0620 00:12:53.881656 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:53.881699 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 00:12:53.882146 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:53.882053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:41202)
0620 00:12:53.882100 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:53.882127 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:12:53.882295 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:53.882359 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:53.882426 7857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:54.561902 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:54.561663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:57933)
0620 00:12:54.561797 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:54.561837 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 00:12:54.561960 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:54.561663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:41679)
0620 00:12:54.561860 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:54.561901 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260620 00:12:54.562178 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:54.561843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:56429)
0620 00:12:54.562090 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:54.562129 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":42,"threads_started":1}
I20260620 00:12:55.562731 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:55.562427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:53573)
0620 00:12:55.562620 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:55.562663 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260620 00:12:55.562747 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:55.562428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:58821)
0620 00:12:55.562664 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:55.562708 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
I20260620 00:12:55.562922 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:55.562659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:59285)
0620 00:12:55.562778 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:55.562868 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
I20260620 00:12:56.563464 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:56.563216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:38311)
0620 00:12:56.563354 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:56.563399 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260620 00:12:56.563592 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:56.563210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:42599)
0620 00:12:56.563516 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:56.563549 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":70,"threads_started":1}
I20260620 00:12:56.563606 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:56.563399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:56263)
0620 00:12:56.563511 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:56.563551 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":47,"threads_started":1}
I20260620 00:12:57.564203 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:57.563957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:46335)
0620 00:12:57.564100 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:57.564144 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260620 00:12:57.564400 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:57.564147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:47907)
0620 00:12:57.564311 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:57.564347 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260620 00:12:57.564522 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:57.563946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:60667)
0620 00:12:57.564384 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:57.564475 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":79,"threads_started":1}
I20260620 00:12:58.496889 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ace22bf7b20425e973c73cde979f0f7)
I20260620 00:12:58.496999 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6ace22bf7b20425e973c73cde979f0f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33771 } } peers { permanent_uuid: "f8cc7382290d4f10bec0f8e0917a5979" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43293 } attrs { promote: false } } peers { permanent_uuid: "3f9a5161bdfd4b35a4b85c50546a0a03" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45695 } attrs { promote: true } }
I20260620 00:12:58.497169 7872 leader_election.cc:288] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771)
I20260620 00:12:58.497677 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:58.497472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:60212)
0620 00:12:58.497596 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:58.497638 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260620 00:12:58.497781 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260620 00:12:58.498268 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:58.498091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.0.1:60216)
0620 00:12:58.498200 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:58.498239 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 00:12:58.498358 7068 leader_election.cc:334] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ace22bf7b20425e973c73cde979f0f7 (127.0.98.62:33771): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
I20260620 00:12:58.498425 7068 leader_election.cc:302] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f8cc7382290d4f10bec0f8e0917a5979; no voters: 6ace22bf7b20425e973c73cde979f0f7
I20260620 00:12:58.498499 7872 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8cc7382290d4f10bec0f8e0917a5979 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:12:58.565229 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:58.564966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:57523)
0620 00:12:58.565148 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:58.565178 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":73,"threads_started":1}
I20260620 00:12:58.565229 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:58.564966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:39793)
0620 00:12:58.565117 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:58.565176 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260620 00:12:58.565358 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:58.565003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:48839)
0620 00:12:58.565264 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:58.565308 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":78,"threads_started":1}
I20260620 00:12:59.566008 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:59.565814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:53629)
0620 00:12:59.565938 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:59.565965 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":44,"threads_started":1}
I20260620 00:12:59.566008 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:59.565672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:51953)
0620 00:12:59.565892 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:59.565939 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":65,"threads_started":1}
I20260620 00:12:59.566039 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:12:59.565761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:46505)
0620 00:12:59.565942 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:12:59.565983 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":59,"threads_started":1}
I20260620 00:13:00.566848 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:00.566565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:35759)
0620 00:13:00.566741 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:00.566786 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260620 00:13:00.567009 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:00.566850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:39873)
0620 00:13:00.566945 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:00.566971 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20260620 00:13:00.566992 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:00.566560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:37227)
0620 00:13:00.566904 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:00.566944 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":72,"threads_started":1}
I20260620 00:13:01.567916 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:01.567490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.1:34449)
0620 00:13:01.567797 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:01.567844 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":71,"threads_started":1}
I20260620 00:13:01.567965 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:01.567537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.3:55957)
0620 00:13:01.567813 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:01.567845 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":63,"threads_started":1}
I20260620 00:13:01.567973 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:01.567512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33771 (local address 127.0.98.2:37325)
0620 00:13:01.567851 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:01.567915 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":64,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:33771,127.0.98.61:43293,127.0.98.60:45695, num_attempts: 1)
I20260620 00:13:01.767297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 7230
I20260620 00:13:01.772387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 7456
I20260620 00:13:01.777760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 7592
I20260620 00:13:01.784066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 6948
I20260620 00:13:01.789340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 7136
2026-06-20T00:13:01Z chronyd exiting
I20260620 00:13:01.805598 392 test_util.cc:175] -----------------------------------------------
I20260620 00:13:01.805682 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914322045185-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33486 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:13:01 dist-test-slave-28bv krb5kdc[7889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:13:01 dist-test-slave-28bv krb5kdc[7889](info): set up 2 sockets
Jun 20 00:13:01 dist-test-slave-28bv krb5kdc[7889](info): commencing operation
krb5kdc: starting...
W20260620 00:13:02.877132 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:13:02 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:13:03.004194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44039
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:03.085111 7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:03.085434 7902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:03.085479 7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:03.086721 7902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:03.086805 7902 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:13:03.086828 7902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:03.086860 7902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:03.086906 7902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:03.086931 7902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:03.088465 7902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44039
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:03.088773 7902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:03.089018 7902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:03.094422 7908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.094478 7907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.094470 7910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.094518 7902 server_base.cc:1034] running on GCE node
I20260620 00:13:03.094965 7902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:03.095402 7902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:03.096626 7902 hybrid_clock.cc:648] HybridClock initialized: now 1781914383096607 us; error 59 us; skew 500 ppm
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:03.099344 7902 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:13:03.099841 7902 webserver.cc:460] Webserver started at http://127.0.98.62:45653/ using document root <none> and password file <none>
I20260620 00:13:03.100085 7902 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:03.100152 7902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:03.100292 7902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:03.101472 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "0e57984e283b4550bfdb90028d404d58"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "aab7fe875b1202df1d26f9c041e060bc"
server_key_iv: "4eaa072121756008cf281ecbb7b170c5"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.101886 7902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e57984e283b4550bfdb90028d404d58"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "aab7fe875b1202df1d26f9c041e060bc"
server_key_iv: "4eaa072121756008cf281ecbb7b170c5"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.103312 7902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:13:03.104157 7917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.104352 7902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:03.104453 7902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "0e57984e283b4550bfdb90028d404d58"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "aab7fe875b1202df1d26f9c041e060bc"
server_key_iv: "4eaa072121756008cf281ecbb7b170c5"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.104611 7902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:03.120398 7902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:03.122742 7902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:03.122867 7902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:03.127796 7902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44039
I20260620 00:13:03.127893 7979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44039 every 8 connection(s)
I20260620 00:13:03.128225 7902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:03.129051 7980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.129407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 7902
I20260620 00:13:03.129515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:13:03.129737 392 external_mini_cluster.cc:1428] Setting key 809dd4ad713828f5370cd3ea6bca4a96
I20260620 00:13:03.132265 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Bootstrap starting.
I20260620 00:13:03.132963 7980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.133339 7980 log.cc:824] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:03.134171 7980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: No bootstrap required, opened a new log
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:03.136902 7980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } }
I20260620 00:13:03.137066 7980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.137127 7980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e57984e283b4550bfdb90028d404d58, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.137271 7980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } }
I20260620 00:13:03.137344 7980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:03.137368 7980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:03.137441 7980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:03.138279 7980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } }
I20260620 00:13:03.138407 7980 leader_election.cc:302] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e57984e283b4550bfdb90028d404d58; no voters:
I20260620 00:13:03.138600 7980 leader_election.cc:288] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:13:03.138629 7985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:03.138772 7985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [term 1 LEADER]: Becoming Leader. State: Replica: 0e57984e283b4550bfdb90028d404d58, State: Running, Role: LEADER
I20260620 00:13:03.138928 7980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:03.138947 7985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } }
I20260620 00:13:03.139422 7987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e57984e283b4550bfdb90028d404d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } } }
I20260620 00:13:03.139688 7987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:03.139607 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e57984e283b4550bfdb90028d404d58. Latest consensus state: current_term: 1 leader_uuid: "0e57984e283b4550bfdb90028d404d58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e57984e283b4550bfdb90028d404d58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44039 } } }
I20260620 00:13:03.140010 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:03.140183 7994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:03.140760 7994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:03.141518 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.130300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.98.62:44039)
0620 00:13:03.130495 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.130505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.130536 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:13:03.130956 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.130963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.131012 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.131137 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.131965 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.133021 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.134090 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.134584 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.134679 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.137657 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.137662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.137662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.137674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.139734 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.140183 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.140186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.140186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.140329 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.140576 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.140577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.140577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.140858 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.141030 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.141149 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.141290 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260620 00:13:03.142382 7994 catalog_manager.cc:1347] Generated new cluster ID: 4263664aca23404f85f11a2861f7c065
I20260620 00:13:03.142441 7994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:13:03.158957 7994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:13:03.159554 7994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:03.164575 7994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Generated new TSK 0
I20260620 00:13:03.164791 7994 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 00:13:03.199553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44039
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:03.291929 8006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:03.292213 8006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:03.292260 8006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:03.293658 8006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:03.293744 8006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:03.293780 8006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:13:03.293823 8006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:03.295492 8006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:03.295801 8006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:03.296103 8006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:03.301688 8014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.301656 8011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.302073 8006 server_base.cc:1034] running on GCE node
W20260620 00:13:03.302064 8012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.302400 8006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:03.302672 8006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:03.303876 8006 hybrid_clock.cc:648] HybridClock initialized: now 1781914383303870 us; error 41 us; skew 500 ppm
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:03.307054 8006 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:13:03.307698 8006 webserver.cc:460] Webserver started at http://127.0.98.1:38521/ using document root <none> and password file <none>
I20260620 00:13:03.308135 8006 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:03.308234 8006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:03.308400 8006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:03.309717 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "00a45219c89249f69c9b31939d7b41ac"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "bde3fba21ea35889320e58a7cb159d09"
server_key_iv: "0211db62e8d80f2f66ae8b946cd74d52"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.310148 8006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00a45219c89249f69c9b31939d7b41ac"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "bde3fba21ea35889320e58a7cb159d09"
server_key_iv: "0211db62e8d80f2f66ae8b946cd74d52"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.311944 8006 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:13:03.313127 8021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.313374 8006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:13:03.313503 8006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "00a45219c89249f69c9b31939d7b41ac"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "bde3fba21ea35889320e58a7cb159d09"
server_key_iv: "0211db62e8d80f2f66ae8b946cd74d52"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.313602 8006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:03.326210 8006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:03.328807 8006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:03.328984 8006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:03.329667 8006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:03.329730 8006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.329768 8006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:03.329792 8006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:03.337596 8006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43715
I20260620 00:13:03.337689 8136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43715 every 8 connection(s)
I20260620 00:13:03.338073 8006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:13:03.340993 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.330178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60541 (local address 127.0.98.62:44039)
0620 00:13:03.330250 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.330253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.330494 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:13:03.331321 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.331322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.331334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.331397 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.332451 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.333033 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.334106 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.334322 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.334398 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.337380 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.337382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.337382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.337392 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.339278 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.339748 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.339749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.339749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.339814 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.340195 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.340197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.340197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.340339 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.340408 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.340833 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.340923 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:13:03.341096 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.330130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.1:60541)
0620 00:13:03.330336 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.330396 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.331189 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.331489 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.331495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.331979 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.332324 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.332332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.333141 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.333142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.334008 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.334011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.334405 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.334407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.334555 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.335198 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.335207 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.337263 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.339382 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.339388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.339389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.339645 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.339956 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.339957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.339957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.340115 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.340447 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.340454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.340662 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.340766 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.340911 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":104,"threads_started":1}
I20260620 00:13:03.341816 8137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44039
I20260620 00:13:03.341926 8137 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:03.342139 8137 heartbeater.cc:511] Master 127.0.98.62:44039 requested a full tablet report, sending...
I20260620 00:13:03.342860 7934 ts_manager.cc:194] Registered new tserver with Master: 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715)
I20260620 00:13:03.343700 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60541
I20260620 00:13:03.346076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8006
I20260620 00:13:03.346221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:13:03.346446 392 external_mini_cluster.cc:1428] Setting key 97c9d188348972a31824728de13fb723
I20260620 00:13:03.350008 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.343100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38167 (local address 127.0.98.62:44039)
0620 00:13:03.343171 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.343173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.343263 (+ 90us) server_negotiation.cc:407] Connection header received
0620 00:13:03.343307 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.343308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.343318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.343383 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.343758 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.344393 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.345305 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.345503 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.345597 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.346800 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.346802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.346802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.346810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.348657 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.349131 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.349133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.349134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.349202 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.349415 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.349416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.349416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.349557 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.349636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.349792 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.349899 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:13:03.350028 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.343060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.1:38167)
0620 00:13:03.343204 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.343213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.343266 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.343423 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.343425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.343619 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.343677 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.343678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.344530 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.344532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.345212 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.345214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.345601 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.345602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.345664 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.346230 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.346235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.346660 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.348756 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.348757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.348757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.348999 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.349273 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.349274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.349274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.349340 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.349626 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.349627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.349684 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.349764 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.349909 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:13:03.381541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44039
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:03.466722 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:03.466967 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:03.466996 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:03.468264 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:03.468313 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:03.468335 8146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:13:03.468354 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:03.469985 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:03.470214 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:03.470407 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:03.475997 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.475991 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.475998 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.476613 8146 server_base.cc:1034] running on GCE node
I20260620 00:13:03.477066 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:03.477398 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:03.478667 8146 hybrid_clock.cc:648] HybridClock initialized: now 1781914383478642 us; error 70 us; skew 500 ppm
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:03.481825 8146 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:13:03.482462 8146 webserver.cc:460] Webserver started at http://127.0.98.2:40175/ using document root <none> and password file <none>
I20260620 00:13:03.482717 8146 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:03.482788 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:03.482945 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:03.484135 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "e989b24be1854d28825b07473e75143f"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "906a6246263bc0c7b26403f8983b4693"
server_key_iv: "1cdf1cfe8c9ab4f9c81efb014e1b18cd"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.484601 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e989b24be1854d28825b07473e75143f"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "906a6246263bc0c7b26403f8983b4693"
server_key_iv: "1cdf1cfe8c9ab4f9c81efb014e1b18cd"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.486186 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:03.487164 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.487488 8146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:03.487761 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "e989b24be1854d28825b07473e75143f"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "906a6246263bc0c7b26403f8983b4693"
server_key_iv: "1cdf1cfe8c9ab4f9c81efb014e1b18cd"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.487856 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:03.503695 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:03.506145 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:03.506399 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:03.507077 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:03.507128 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.507160 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:03.507200 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:03.514652 8146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34523
I20260620 00:13:03.515012 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34523 every 8 connection(s)
I20260620 00:13:03.515079 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:13:03.517109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8146
I20260620 00:13:03.517238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:13:03.517457 392 external_mini_cluster.cc:1428] Setting key ba40486c0c11eaed984e29d2b2116cb9
I20260620 00:13:03.517493 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.507519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59755 (local address 127.0.98.62:44039)
0620 00:13:03.507584 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.507586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.507823 (+ 237us) server_negotiation.cc:407] Connection header received
0620 00:13:03.508577 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.508578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.508593 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.508657 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.509312 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.509856 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.510950 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.511167 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.511242 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.514269 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.514271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.514271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.514279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.515891 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.516271 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.516271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.516272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.516321 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.516707 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.516708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.516708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.516826 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.516898 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.517261 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.517356 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:13:03.517702 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.507516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.2:59755)
0620 00:13:03.507706 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.507748 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.508438 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.508736 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.508740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.509007 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.509210 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.509214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.509959 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.509961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.510846 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.510849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.511262 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.511263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.511423 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.512003 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.512013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.514170 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.515973 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.515979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.515981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.516204 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.516375 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.516375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.516376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.516610 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.516910 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.516916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.517095 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.517237 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.517434 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260620 00:13:03.518189 8277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44039
I20260620 00:13:03.518322 8277 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:03.518554 8277 heartbeater.cc:511] Master 127.0.98.62:44039 requested a full tablet report, sending...
I20260620 00:13:03.518956 7934 ts_manager.cc:194] Registered new tserver with Master: e989b24be1854d28825b07473e75143f (127.0.98.2:34523)
I20260620 00:13:03.519481 7934 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59755
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:13:03.526605 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.519939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48133 (local address 127.0.98.62:44039)
0620 00:13:03.519978 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.519980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.520006 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:13:03.520072 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.520073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.520084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.520146 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.520513 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.521145 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.522414 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.522604 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.522661 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.523851 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.523853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.523853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.523859 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.525376 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.525790 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.525790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.525791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.525839 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.526052 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.526052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.526052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.526189 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.526243 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.526448 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.526539 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:13:03.526683 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.519751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.2:48133)
0620 00:13:03.519949 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.519960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.520020 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.520148 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.520149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.520345 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.520425 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.520427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.521266 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.521267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.522274 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.522277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.522682 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.522683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.522766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.523252 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.523258 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.523744 (+ 486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.525462 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.525464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.525464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.525706 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.525902 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.525903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.525904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.525977 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.526253 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.526254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.526305 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.526400 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.526526 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":60,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:13:03.544759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44039
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:03.630803 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:03.631042 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:03.631068 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:03.632234 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:03.632272 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:03.632292 8286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:13:03.632309 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:03.633774 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:03.634034 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:03.634308 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:03.639674 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:03.639691 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.639853 8286 server_base.cc:1034] running on GCE node
W20260620 00:13:03.639748 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:03.640166 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:03.640465 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:03.641681 8286 hybrid_clock.cc:648] HybridClock initialized: now 1781914383641655 us; error 36 us; skew 500 ppm
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:03.644320 8286 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:13:03.644822 8286 webserver.cc:460] Webserver started at http://127.0.98.3:39351/ using document root <none> and password file <none>
I20260620 00:13:03.645056 8286 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:03.645129 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:03.645279 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:03.646353 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "09da830805e24056a7fe22ad92d3aef9"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "f045f1ea7035f1e28f454ebb5b6c6175"
server_key_iv: "76acfc904e3dc7d93e6c56c0ccde23a6"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.646715 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "09da830805e24056a7fe22ad92d3aef9"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "f045f1ea7035f1e28f454ebb5b6c6175"
server_key_iv: "76acfc904e3dc7d93e6c56c0ccde23a6"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.648061 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:13:03.648900 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.649103 8286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:03.649201 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "09da830805e24056a7fe22ad92d3aef9"
format_stamp: "Formatted at 2026-06-20 00:13:03 on dist-test-slave-28bv"
server_key: "f045f1ea7035f1e28f454ebb5b6c6175"
server_key_iv: "76acfc904e3dc7d93e6c56c0ccde23a6"
server_key_version: "encryptionkey@0"
I20260620 00:13:03.649274 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:03.663085 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:03.665637 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:03.665800 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:03.666370 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:03.666440 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.666512 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:03.666544 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.673623 8286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41327
I20260620 00:13:03.673692 8416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41327 every 8 connection(s)
I20260620 00:13:03.674042 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:03.677906 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.666883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59125 (local address 127.0.98.62:44039)
0620 00:13:03.666938 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.666941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.667167 (+ 226us) server_negotiation.cc:407] Connection header received
0620 00:13:03.667703 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.667705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.667717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.667776 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.668813 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.669742 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.671519 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.671840 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.671936 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.674791 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.674794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.674794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.674803 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.676358 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.676888 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.676890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.676890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.676957 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.677220 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.677221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.677221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.677355 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.677428 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.677721 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.677809 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:13:03.678125 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.666876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.3:59125)
0620 00:13:03.667024 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.667065 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.667596 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.667894 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.667899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.668349 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.668704 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.668711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.669854 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.669855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.671408 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.671412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.671948 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.671949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.672096 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.672704 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.672715 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.674680 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.676485 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.676493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.676495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.676798 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.677024 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.677025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.677025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.677152 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.677418 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.677425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.677575 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.677687 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.677862 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260620 00:13:03.678610 8417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44039
I20260620 00:13:03.678725 8417 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:03.678910 8417 heartbeater.cc:511] Master 127.0.98.62:44039 requested a full tablet report, sending...
I20260620 00:13:03.679352 7933 ts_manager.cc:194] Registered new tserver with Master: 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327)
I20260620 00:13:03.679854 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59125
I20260620 00:13:03.679904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8286
I20260620 00:13:03.680001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:13:03.680183 392 external_mini_cluster.cc:1428] Setting key da6fdbc05a1fdbc8a56f649171464b5f
I20260620 00:13:03.681051 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:13:03.686877 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.679752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36753 (local address 127.0.98.62:44039)
0620 00:13:03.679794 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.679796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.679941 (+ 145us) server_negotiation.cc:407] Connection header received
0620 00:13:03.680021 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.680022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.680032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.680063 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.680596 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.681286 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.682012 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.682336 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.682408 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.683534 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.683535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.683536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.683542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.685503 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.685944 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.685946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.685946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.686014 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.686226 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.686227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.686227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.686385 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.686473 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.686725 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.686809 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:13:03.687004 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.679692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.98.3:36753)
0620 00:13:03.679861 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.679872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.679939 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.680136 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.680137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.680400 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:03.680480 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.680483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.681371 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.681372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.681925 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.681927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.682411 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.682412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.682459 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.682989 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:03.682995 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:03.683449 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:03.685624 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.685625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.685626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.685861 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.686101 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:03.686102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:03.686102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:03.686161 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:03.686507 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:03.686508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:03.686558 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.686682 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.686833 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:13:03.706684 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:03.700737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.0.98.62:44039)
0620 00:13:03.700805 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.700807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.700920 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:13:03.701058 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.701059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.701070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.701143 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.701775 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.702281 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.703051 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.703243 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.703327 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.706580 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56356: BlockingRecv error: recv got EOF from 127.0.0.1:56356 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 00:13:03 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:03.714982 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.707744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.62:44039)
0620 00:13:03.707812 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.707814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.707875 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:13:03.707964 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.707965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.707976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.708095 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:03.708697 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.709188 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.709937 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.710143 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.710221 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.712678 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:03.712680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:03.712680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:03.712687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:03.713960 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.714311 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.714312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.714312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.714363 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:03.714521 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:03.714521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:03.714522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:03.714649 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:03.714726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.714855 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.714913 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:13:03.716747 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:13:03.718350 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:03.725106 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.720771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.0.1:42758)
0620 00:13:03.721129 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.721138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.721210 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.721443 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.721446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.721448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:03.721519 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.721521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.722605 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.722606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.723930 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.723932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.724769 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.724853 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.724867 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.724947 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.725005 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":5,"thread_start_us":91,"threads_started":1}
I20260620 00:13:03.725111 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.720764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43715 (local address 127.0.0.1:41582)
0620 00:13:03.720888 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.720930 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.720993 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.721554 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.721555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.721557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:03.721621 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.721623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.722930 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.722931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.723970 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.723972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.724927 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.724985 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.724988 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.725028 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.725057 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":80,"threads_started":1}
I20260620 00:13:03.725359 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.720990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41327 (local address 127.0.0.1:52390)
0620 00:13:03.721674 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.721684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.721730 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.721893 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.721894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.721896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:03.721958 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.721960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.723143 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.723144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.724109 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.724111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.725137 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.725138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.725206 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.725210 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.725259 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.725300 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":83,"threads_started":1}
I20260620 00:13:03.725476 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.720870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.0.98.2:34523)
0620 00:13:03.721040 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.721045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.721200 (+ 155us) server_negotiation.cc:407] Connection header received
0620 00:13:03.721279 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.721282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.721307 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.721422 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:03.721614 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.722517 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724034 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.724674 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724768 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.724854 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.725326 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.725368 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260620 00:13:03.725623 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.721187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.98.3:41327)
0620 00:13:03.721364 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.721368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.721736 (+ 368us) server_negotiation.cc:407] Connection header received
0620 00:13:03.721789 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.721792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.721808 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.721886 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:03.722025 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.723063 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724190 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.724881 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724942 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.725004 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.725469 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.725517 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260620 00:13:03.725646 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.721211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.98.1:43715)
0620 00:13:03.721368 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.721373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.721382 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:03.721460 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.721463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.721479 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.721554 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:03.721712 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.722846 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724066 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.724840 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.724930 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.725006 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.725044 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.725080 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260620 00:13:03.726231 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 (DEFAULT_TABLE table=test-table [id=a1279b7500864bc59d1970eb8902cb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.726235 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 (DEFAULT_TABLE table=test-table [id=a1279b7500864bc59d1970eb8902cb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.726317 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 (DEFAULT_TABLE table=test-table [id=a1279b7500864bc59d1970eb8902cb31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.726609 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952b4580ebdf46499f7fd007c8f9ffd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.726609 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952b4580ebdf46499f7fd007c8f9ffd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.726642 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 952b4580ebdf46499f7fd007c8f9ffd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.729552 8445 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Bootstrap starting.
I20260620 00:13:03.730130 8445 tablet_bootstrap.cc:654] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.730398 8446 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Bootstrap starting.
I20260620 00:13:03.730499 8445 log.cc:824] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:03.730887 8447 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Bootstrap starting.
I20260620 00:13:03.731070 8446 tablet_bootstrap.cc:654] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.731321 8445 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: No bootstrap required, opened a new log
I20260620 00:13:03.731391 8446 log.cc:824] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:03.731448 8445 ts_tablet_manager.cc:1397] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:03.731503 8447 tablet_bootstrap.cc:654] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.731825 8447 log.cc:824] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:03.732170 8446 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: No bootstrap required, opened a new log
I20260620 00:13:03.732249 8446 ts_tablet_manager.cc:1397] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:13:03.732666 8447 tablet_bootstrap.cc:492] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: No bootstrap required, opened a new log
I20260620 00:13:03.732738 8447 ts_tablet_manager.cc:1397] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:13:03.733763 8446 raft_consensus.cc:348] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.733804 8445 raft_consensus.cc:348] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.733922 8446 raft_consensus.cc:374] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.733955 8445 raft_consensus.cc:374] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.733989 8446 raft_consensus.cc:729] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a45219c89249f69c9b31939d7b41ac, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.734025 8445 raft_consensus.cc:729] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.734126 8446 consensus_queue.cc:260] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.734163 8445 consensus_queue.cc:260] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.734421 8445 ts_tablet_manager.cc:1428] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:13:03.734447 8446 ts_tablet_manager.cc:1428] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:03.734455 8137 heartbeater.cc:503] Master 127.0.98.62:44039 was elected leader, sending a full tablet report...
I20260620 00:13:03.734609 8417 heartbeater.cc:503] Master 127.0.98.62:44039 was elected leader, sending a full tablet report...
I20260620 00:13:03.734817 8447 raft_consensus.cc:348] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.735059 8447 raft_consensus.cc:374] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.735142 8447 raft_consensus.cc:729] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989b24be1854d28825b07473e75143f, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.735283 8447 consensus_queue.cc:260] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.735620 8447 ts_tablet_manager.cc:1428] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:13:03.735673 8277 heartbeater.cc:503] Master 127.0.98.62:44039 was elected leader, sending a full tablet report...
W20260620 00:13:03.765691 8278 tablet.cc:2369] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:13:03.800889 8451 raft_consensus.cc:482] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:03.801059 8451 raft_consensus.cc:504] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.801540 8451 leader_election.cc:288] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e989b24be1854d28825b07473e75143f (127.0.98.2:34523), 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715)
I20260620 00:13:03.804667 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.801870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46641 (local address 127.0.98.2:34523)
0620 00:13:03.801919 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.801922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.801945 (+ 23us) server_negotiation.cc:407] Connection header received
0620 00:13:03.801975 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.801976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.801988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.802090 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:03.802313 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.802887 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.803885 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.804276 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.804330 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.804360 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.804520 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.804593 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:13:03.804674 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.802027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55807 (local address 127.0.98.1:43715)
0620 00:13:03.802075 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.802078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.802241 (+ 163us) server_negotiation.cc:407] Connection header received
0620 00:13:03.802303 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.802305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.802317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.802368 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:03.802514 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.802975 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.803924 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.804287 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.804371 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.804381 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.804574 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.804608 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:13:03.804674 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.801723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.98.3:46641)
0620 00:13:03.801780 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.801796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.801865 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.802104 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.802105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.802107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:03.802239 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.802241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.802974 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.802975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.803793 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.803795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.804355 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.804356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.804445 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.804449 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.804498 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.804592 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:13:03.804685 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.801799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43715 (local address 127.0.98.3:55807)
0620 00:13:03.802155 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:03.802165 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:03.802230 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:03.802387 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:03.802388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:03.802390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:03.802453 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.802455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.803099 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.803100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.803817 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:03.803818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.804389 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.804390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:03.804459 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.804462 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:13:03.804532 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:13:03.804594 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":9,"thread_start_us":177,"threads_started":1}
I20260620 00:13:03.805120 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952b4580ebdf46499f7fd007c8f9ffd6" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac" is_pre_election: true
I20260620 00:13:03.805284 8091 raft_consensus.cc:2454] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:03.805243 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952b4580ebdf46499f7fd007c8f9ffd6" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f" is_pre_election: true
I20260620 00:13:03.805401 8230 raft_consensus.cc:2454] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:03.805568 8303 leader_election.cc:302] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, 09da830805e24056a7fe22ad92d3aef9; no voters:
I20260620 00:13:03.805698 8451 raft_consensus.cc:2781] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:03.805779 8451 raft_consensus.cc:482] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:03.805814 8451 raft_consensus.cc:3037] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:03.806921 8451 raft_consensus.cc:504] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.807089 8451 leader_election.cc:288] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Requested vote from peers e989b24be1854d28825b07473e75143f (127.0.98.2:34523), 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715)
I20260620 00:13:03.807253 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952b4580ebdf46499f7fd007c8f9ffd6" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac"
I20260620 00:13:03.807264 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "952b4580ebdf46499f7fd007c8f9ffd6" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f"
I20260620 00:13:03.807335 8091 raft_consensus.cc:3037] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:03.807341 8230 raft_consensus.cc:3037] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:03.808173 8230 raft_consensus.cc:2454] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:03.808182 8091 raft_consensus.cc:2454] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:03.808341 8305 leader_election.cc:302] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09da830805e24056a7fe22ad92d3aef9, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:03.808440 8451 raft_consensus.cc:2781] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:03.808611 8451 raft_consensus.cc:686] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Becoming Leader. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Running, Role: LEADER
I20260620 00:13:03.808720 8451 consensus_queue.cc:237] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.809522 7934 catalog_manager.cc:5495] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "09da830805e24056a7fe22ad92d3aef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } health_report { overall_health: HEALTHY } } }
I20260620 00:13:03.820209 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:03.817684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.98.3:41327)
0620 00:13:03.817736 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:03.817738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:03.817748 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:13:03.817832 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:03.817833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:03.817856 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:03.817937 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:03.818070 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.818615 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.819445 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:03.819642 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:03.819709 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:03.819880 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:03.819985 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:03.820079 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:03.820097 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 00:13:03.820116 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:13:03.822129 8091 raft_consensus.cc:1261] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:03.822165 8230 raft_consensus.cc:1261] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:03.822516 8455 consensus_queue.cc:1035] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:03.822655 8451 consensus_queue.cc:1035] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:03.825026 8458 mvcc.cc:205] Tried to move back new op lower bound from 7298721316132253696 to 7298721316081123328. Current Snapshot: MvccSnapshot[applied={T|T < 7298721316132253696}]
I20260620 00:13:03.825798 8460 mvcc.cc:205] Tried to move back new op lower bound from 7298721316132253696 to 7298721316081123328. Current Snapshot: MvccSnapshot[applied={T|T < 7298721316132253696}]
I20260620 00:13:03.827863 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:03.827986 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:03.828748 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Sending DeleteTablet for 3 replicas of tablet 952b4580ebdf46499f7fd007c8f9ffd6
I20260620 00:13:03.829069 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42758
I20260620 00:13:03.829118 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41582
I20260620 00:13:03.829355 8469 tablet_replica.cc:330] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: stopping tablet replica
I20260620 00:13:03.829355 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 952b4580ebdf46499f7fd007c8f9ffd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52390
I20260620 00:13:03.829480 8469 raft_consensus.cc:2229] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:03.829569 8470 tablet_replica.cc:330] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: stopping tablet replica
I20260620 00:13:03.829694 8470 raft_consensus.cc:2229] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:13:03.829748 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 00:13:03.829833 8470 raft_consensus.cc:2258] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 00:13:03.829897 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:03.829982 8468 tablet_replica.cc:330] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: stopping tablet replica
I20260620 00:13:03.830080 8468 raft_consensus.cc:2229] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:03.830333 8468 raft_consensus.cc:2258] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:03.830778 8468 ts_tablet_manager.cc:1905] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:03.831530 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 1817a1423478487ba3b98c192e68c2c5 (DEFAULT_TABLE table=test-table [id=fbc7186ac2184404b70ebb32b794fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.831648 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1817a1423478487ba3b98c192e68c2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.832667 8468 ts_tablet_manager.cc:1918] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:03.832736 8468 log.cc:1196] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/952b4580ebdf46499f7fd007c8f9ffd6
I20260620 00:13:03.832705 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 1817a1423478487ba3b98c192e68c2c5 (DEFAULT_TABLE table=test-table [id=fbc7186ac2184404b70ebb32b794fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.832826 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1817a1423478487ba3b98c192e68c2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.833005 8468 ts_tablet_manager.cc:1939] T 952b4580ebdf46499f7fd007c8f9ffd6 P e989b24be1854d28825b07473e75143f: Deleting consensus metadata
I20260620 00:13:03.833122 8445 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Bootstrap starting.
I20260620 00:13:03.833573 8445 tablet_bootstrap.cc:654] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.833575 7921 catalog_manager.cc:4841] TS e989b24be1854d28825b07473e75143f (127.0.98.2:34523): tablet 952b4580ebdf46499f7fd007c8f9ffd6 (table test-table [id=a1279b7500864bc59d1970eb8902cb31]) successfully deleted
I20260620 00:13:03.834017 8447 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Bootstrap starting.
I20260620 00:13:03.829680 8469 raft_consensus.cc:2258] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:03.831470 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 1817a1423478487ba3b98c192e68c2c5 (DEFAULT_TABLE table=test-table [id=fbc7186ac2184404b70ebb32b794fd04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:03.834393 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1817a1423478487ba3b98c192e68c2c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:03.834417 8447 tablet_bootstrap.cc:654] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.834959 8447 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: No bootstrap required, opened a new log
I20260620 00:13:03.835008 8447 ts_tablet_manager.cc:1397] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:03.835089 8445 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: No bootstrap required, opened a new log
I20260620 00:13:03.835129 8445 ts_tablet_manager.cc:1397] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:13:03.835165 8447 raft_consensus.cc:348] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.835235 8447 raft_consensus.cc:374] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.835258 8447 raft_consensus.cc:729] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989b24be1854d28825b07473e75143f, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.835251 8445 raft_consensus.cc:348] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.835318 8445 raft_consensus.cc:374] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.835297 8447 consensus_queue.cc:260] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.835347 8445 raft_consensus.cc:729] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.835390 8447 ts_tablet_manager.cc:1428] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.835390 8445 consensus_queue.cc:260] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.835469 8445 ts_tablet_manager.cc:1428] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:03.835767 8446 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Bootstrap starting.
I20260620 00:13:03.836030 8470 ts_tablet_manager.cc:1905] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:03.836448 8446 tablet_bootstrap.cc:654] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:03.837055 8469 ts_tablet_manager.cc:1905] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:03.837083 8446 tablet_bootstrap.cc:492] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: No bootstrap required, opened a new log
I20260620 00:13:03.837183 8446 ts_tablet_manager.cc:1397] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:03.837332 8446 raft_consensus.cc:348] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.837405 8446 raft_consensus.cc:374] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:03.837424 8446 raft_consensus.cc:729] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a45219c89249f69c9b31939d7b41ac, State: Initialized, Role: FOLLOWER
I20260620 00:13:03.837476 8446 consensus_queue.cc:260] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:03.837556 8446 ts_tablet_manager.cc:1428] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:03.837669 8470 ts_tablet_manager.cc:1918] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:03.837741 8470 log.cc:1196] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/952b4580ebdf46499f7fd007c8f9ffd6
I20260620 00:13:03.837989 8470 ts_tablet_manager.cc:1939] T 952b4580ebdf46499f7fd007c8f9ffd6 P 09da830805e24056a7fe22ad92d3aef9: Deleting consensus metadata
I20260620 00:13:03.838377 7919 catalog_manager.cc:4841] TS 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327): tablet 952b4580ebdf46499f7fd007c8f9ffd6 (table test-table [id=a1279b7500864bc59d1970eb8902cb31]) successfully deleted
I20260620 00:13:03.838702 8469 ts_tablet_manager.cc:1918] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:03.838785 8469 log.cc:1196] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/952b4580ebdf46499f7fd007c8f9ffd6
I20260620 00:13:03.839051 8469 ts_tablet_manager.cc:1939] T 952b4580ebdf46499f7fd007c8f9ffd6 P 00a45219c89249f69c9b31939d7b41ac: Deleting consensus metadata
W20260620 00:13:03.839207 8138 tablet.cc:2369] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:13:03.839453 7919 catalog_manager.cc:4841] TS 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715): tablet 952b4580ebdf46499f7fd007c8f9ffd6 (table test-table [id=a1279b7500864bc59d1970eb8902cb31]) successfully deleted
W20260620 00:13:03.924691 8418 tablet.cc:2369] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:13:04.002380 8465 raft_consensus.cc:482] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:04.002490 8465 raft_consensus.cc:504] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.002664 8465 leader_election.cc:288] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), e989b24be1854d28825b07473e75143f (127.0.98.2:34523)
I20260620 00:13:04.002929 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1817a1423478487ba3b98c192e68c2c5" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac" is_pre_election: true
I20260620 00:13:04.003043 8091 raft_consensus.cc:2454] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:04.002940 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1817a1423478487ba3b98c192e68c2c5" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f" is_pre_election: true
I20260620 00:13:04.003103 8230 raft_consensus.cc:2454] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:04.003232 8303 leader_election.cc:302] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, 09da830805e24056a7fe22ad92d3aef9; no voters:
I20260620 00:13:04.003343 8465 raft_consensus.cc:2781] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:04.003391 8465 raft_consensus.cc:482] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:04.003407 8465 raft_consensus.cc:3037] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.004094 8465 raft_consensus.cc:504] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.004256 8465 leader_election.cc:288] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Requested vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), e989b24be1854d28825b07473e75143f (127.0.98.2:34523)
I20260620 00:13:04.004411 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1817a1423478487ba3b98c192e68c2c5" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac"
I20260620 00:13:04.004436 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1817a1423478487ba3b98c192e68c2c5" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f"
I20260620 00:13:04.004521 8091 raft_consensus.cc:3037] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.004519 8230 raft_consensus.cc:3037] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.005173 8230 raft_consensus.cc:2454] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:04.005312 8091 raft_consensus.cc:2454] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:04.005352 8305 leader_election.cc:302] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09da830805e24056a7fe22ad92d3aef9, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:04.005460 8465 raft_consensus.cc:2781] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:04.005524 8465 raft_consensus.cc:686] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Becoming Leader. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Running, Role: LEADER
I20260620 00:13:04.005584 8465 consensus_queue.cc:237] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.006313 7933 catalog_manager.cc:5495] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "09da830805e24056a7fe22ad92d3aef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } health_report { overall_health: HEALTHY } } }
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914383, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:04.044873 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.037211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44039 (local address 127.0.0.1:56378)
0620 00:13:04.037395 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.037406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.037487 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.037651 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.037652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.037958 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:04.038001 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.038003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.038678 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.038679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.039200 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.039201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.039598 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.039599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.039713 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.040278 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:04.040282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:04.041814 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:04.043686 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:04.043693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:04.043695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:04.043922 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:04.044124 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:04.044124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:04.044125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:04.044190 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:04.044502 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:04.044509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:04.044612 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.044676 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.044772 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260620 00:13:04.044937 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.037275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.0.98.62:44039)
0620 00:13:04.037361 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.037363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.037467 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:13:04.037578 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.037579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.037588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.037642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:04.038090 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.038607 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.039315 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.039512 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.039598 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.041911 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:04.041913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:04.041913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:04.041920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:04.043467 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:04.044006 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:04.044007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:04.044008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:04.044060 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:04.044303 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:04.044304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:04.044305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:04.044423 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:04.044486 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.044695 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.044819 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":55}
I20260620 00:13:04.047080 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56378:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:13:04.047305 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:04.048861 8071 tablet_service.cc:1467] Processing CreateTablet for tablet b58746ceb14349eab99fc31d50056037 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54051bd1961e4e60a5c66488d2502ebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:13:04.048851 8211 tablet_service.cc:1467] Processing CreateTablet for tablet b58746ceb14349eab99fc31d50056037 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54051bd1961e4e60a5c66488d2502ebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:13:04.048892 8348 tablet_service.cc:1467] Processing CreateTablet for tablet b58746ceb14349eab99fc31d50056037 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54051bd1961e4e60a5c66488d2502ebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:13:04.049018 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58746ceb14349eab99fc31d50056037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.049016 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58746ceb14349eab99fc31d50056037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.049048 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b58746ceb14349eab99fc31d50056037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.050351 8445 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: Bootstrap starting.
I20260620 00:13:04.050532 8446 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: Bootstrap starting.
I20260620 00:13:04.050555 8447 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: Bootstrap starting.
I20260620 00:13:04.050724 8445 tablet_bootstrap.cc:654] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.050921 8446 tablet_bootstrap.cc:654] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.050994 8447 tablet_bootstrap.cc:654] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.051544 8445 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: No bootstrap required, opened a new log
I20260620 00:13:04.051569 8446 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: No bootstrap required, opened a new log
I20260620 00:13:04.051595 8445 ts_tablet_manager.cc:1397] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:04.051610 8446 ts_tablet_manager.cc:1397] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:04.051708 8445 raft_consensus.cc:348] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.051779 8445 raft_consensus.cc:374] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.051754 8446 raft_consensus.cc:348] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.051798 8445 raft_consensus.cc:729] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.051817 8446 raft_consensus.cc:374] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.051839 8446 raft_consensus.cc:729] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a45219c89249f69c9b31939d7b41ac, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.051847 8447 tablet_bootstrap.cc:492] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: No bootstrap required, opened a new log
I20260620 00:13:04.051839 8445 consensus_queue.cc:260] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.051891 8447 ts_tablet_manager.cc:1397] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:04.051879 8446 consensus_queue.cc:260] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.051937 8445 ts_tablet_manager.cc:1428] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:04.051964 8446 ts_tablet_manager.cc:1428] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:04.051963 8465 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.052057 8465 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.052060 8447 raft_consensus.cc:348] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.052088 8452 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.052129 8447 raft_consensus.cc:374] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.052142 8452 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.052151 8447 raft_consensus.cc:729] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989b24be1854d28825b07473e75143f, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.052193 8447 consensus_queue.cc:260] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.052289 8447 ts_tablet_manager.cc:1428] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:04.052330 8453 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.052424 8453 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.110926 8453 raft_consensus.cc:482] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:04.111028 8453 raft_consensus.cc:504] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.111318 8453 leader_election.cc:288] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327)
I20260620 00:13:04.114270 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.111587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34187 (local address 127.0.98.1:43715)
0620 00:13:04.111662 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.111664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.111744 (+ 80us) server_negotiation.cc:407] Connection header received
0620 00:13:04.111810 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.111811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.111820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.111893 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:04.112046 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.112519 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.113547 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.113906 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.113989 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.113999 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.114163 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.114200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:13:04.114284 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.111541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43715 (local address 127.0.98.2:34187)
0620 00:13:04.111670 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.111682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.111751 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.111900 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.111901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.111904 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:04.111968 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.111970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.112620 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.112622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.113453 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.113456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.113987 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.113988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.114060 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.114064 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.114150 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.114195 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260620 00:13:04.114360 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.111826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52547 (local address 127.0.98.3:41327)
0620 00:13:04.111887 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.111890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.111899 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:04.111934 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.111935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.111944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.112008 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:04.112155 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.112680 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.113637 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.114004 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.114084 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.114093 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.114229 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.114278 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:13:04.114382 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.111736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41327 (local address 127.0.98.2:52547)
0620 00:13:04.111836 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.111845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.111887 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.112024 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.112025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.112027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:04.112092 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.112093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.112757 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.112758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.113543 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.113545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.114084 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.114085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.114157 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.114161 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.114224 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.114267 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":41,"threads_started":1}
I20260620 00:13:04.114564 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58746ceb14349eab99fc31d50056037" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac" is_pre_election: true
I20260620 00:13:04.114674 8091 raft_consensus.cc:2454] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 0.
I20260620 00:13:04.114828 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58746ceb14349eab99fc31d50056037" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da830805e24056a7fe22ad92d3aef9" is_pre_election: true
I20260620 00:13:04.114881 8163 leader_election.cc:302] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:04.114946 8371 raft_consensus.cc:2454] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 0.
I20260620 00:13:04.115039 8453 raft_consensus.cc:2781] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:04.115083 8453 raft_consensus.cc:482] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:04.115098 8453 raft_consensus.cc:3037] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.115754 8453 raft_consensus.cc:504] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.115895 8453 leader_election.cc:288] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 election: Requested vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327)
I20260620 00:13:04.116112 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58746ceb14349eab99fc31d50056037" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac"
I20260620 00:13:04.116107 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b58746ceb14349eab99fc31d50056037" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da830805e24056a7fe22ad92d3aef9"
I20260620 00:13:04.116214 8091 raft_consensus.cc:3037] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.116211 8371 raft_consensus.cc:3037] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.116951 8091 raft_consensus.cc:2454] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 1.
I20260620 00:13:04.116977 8371 raft_consensus.cc:2454] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 1.
I20260620 00:13:04.117120 8163 leader_election.cc:302] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09da830805e24056a7fe22ad92d3aef9, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:04.117235 8453 raft_consensus.cc:2781] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:04.117390 8453 raft_consensus.cc:686] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [term 1 LEADER]: Becoming Leader. State: Replica: e989b24be1854d28825b07473e75143f, State: Running, Role: LEADER
I20260620 00:13:04.117465 8453 consensus_queue.cc:237] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.117831 8486 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: TxnStatusTablet state changed. Reason: New leader e989b24be1854d28825b07473e75143f. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.117964 8486 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:13:04.118152 8488 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:13:04.118258 8488 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:13:04.118306 7933 catalog_manager.cc:5495] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f reported cstate change: term changed from 0 to 1, leader changed from <none> to e989b24be1854d28825b07473e75143f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } health_report { overall_health: UNKNOWN } } }
I20260620 00:13:04.127888 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.125355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.0.1:42764)
0620 00:13:04.125427 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.125439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.125488 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.125692 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.125693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.125694 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:13:04.125745 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.125747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.126384 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.126386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.126960 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.126962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.127328 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.127329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.127400 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.127413 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.127714 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.127719 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.127775 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.127823 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:13:04.127873 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.125432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.98.2:34523)
0620 00:13:04.125482 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.125484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.125494 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:13:04.125565 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.125566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.125585 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.125649 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.125813 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.126296 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.127044 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.127245 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.127328 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.127502 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.127618 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.127692 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.127779 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.127800 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:13:04.129603 8371 raft_consensus.cc:1261] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Refusing update from remote peer e989b24be1854d28825b07473e75143f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.129586 8091 raft_consensus.cc:1261] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Refusing update from remote peer e989b24be1854d28825b07473e75143f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.129863 8453 consensus_queue.cc:1035] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.130098 8486 consensus_queue.cc:1035] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.130540 8452 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: TxnStatusTablet state changed. Reason: New leader e989b24be1854d28825b07473e75143f. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.130576 8465 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: TxnStatusTablet state changed. Reason: New leader e989b24be1854d28825b07473e75143f. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.130649 8452 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.130666 8465 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.131441 8453 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.131529 8453 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:13:04.131776 8452 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.131871 8452 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 00a45219c89249f69c9b31939d7b41ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.137188 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.134921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55927 (local address 127.0.98.2:34523)
0620 00:13:04.134965 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.134967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.134979 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:04.135052 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.135053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.135063 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.135101 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.135224 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.135693 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.136445 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.136727 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.136798 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.136957 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.136992 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.137040 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.137101 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.137122 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:13:04.137216 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.134858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.98.3:55927)
0620 00:13:04.134936 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.134950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.134994 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.135105 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.135106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.135107 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:13:04.135154 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.135155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.135773 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.135774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.136354 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.136356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.136795 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.136797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.136856 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.136869 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.137046 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.137047 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.137097 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.137144 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:13:04.139324 8091 raft_consensus.cc:1261] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.139336 8230 raft_consensus.cc:1261] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.139470 8453 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.139561 8453 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P e989b24be1854d28825b07473e75143f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:13:04.139529 8451 consensus_queue.cc:1035] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.139639 8465 consensus_queue.cc:1035] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.140434 8451 tablet_replica.cc:439] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } } }
I20260620 00:13:04.140518 8451 tablet_replica.cc:442] T b58746ceb14349eab99fc31d50056037 P 09da830805e24056a7fe22ad92d3aef9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:13:04.145875 8459 mvcc.cc:205] Tried to move back new op lower bound from 7298721317432995840 to 7298721316887199744. Current Snapshot: MvccSnapshot[applied={T|T < 7298721317432995840}]
I20260620 00:13:04.148317 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.145113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35481 (local address 127.0.98.3:41327)
0620 00:13:04.145160 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.145162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.145301 (+ 139us) server_negotiation.cc:407] Connection header received
0620 00:13:04.145375 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.145376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.145389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.145434 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.145593 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.146268 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.147480 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.147787 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.147865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.148065 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.148101 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.148173 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.148216 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.148240 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:13:04.148329 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.145058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41327 (local address 127.0.98.2:35481)
0620 00:13:04.145223 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.145235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.145300 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.145443 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.145444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.145446 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:13:04.145517 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.145519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.146424 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.146432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.147413 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.147416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.147900 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.147901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.147957 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.147980 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.148177 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.148179 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.148213 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.148249 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260620 00:13:04.156884 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.157001 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.157737 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Sending DeleteTablet for 3 replicas of tablet 1817a1423478487ba3b98c192e68c2c5
I20260620 00:13:04.157994 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 1817a1423478487ba3b98c192e68c2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41582
I20260620 00:13:04.158113 8469 tablet_replica.cc:330] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: stopping tablet replica
I20260620 00:13:04.158171 8469 raft_consensus.cc:2229] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.158210 8469 raft_consensus.cc:2258] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.158347 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 1817a1423478487ba3b98c192e68c2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52390
I20260620 00:13:04.158468 8470 tablet_replica.cc:330] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: stopping tablet replica
I20260620 00:13:04.158526 8470 raft_consensus.cc:2229] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:13:04.158609 8470 raft_consensus.cc:2258] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.158764 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 1817a1423478487ba3b98c192e68c2c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42758
I20260620 00:13:04.158874 8468 tablet_replica.cc:330] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: stopping tablet replica
I20260620 00:13:04.158933 8468 raft_consensus.cc:2229] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.158982 8468 raft_consensus.cc:2258] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.159405 8468 ts_tablet_manager.cc:1905] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.161584 8468 ts_tablet_manager.cc:1918] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:13:04.161654 8468 log.cc:1196] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/1817a1423478487ba3b98c192e68c2c5
I20260620 00:13:04.161672 8469 ts_tablet_manager.cc:1905] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.161767 8470 ts_tablet_manager.cc:1905] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.161909 8468 ts_tablet_manager.cc:1939] T 1817a1423478487ba3b98c192e68c2c5 P e989b24be1854d28825b07473e75143f: Deleting consensus metadata
I20260620 00:13:04.163480 8470 ts_tablet_manager.cc:1918] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:13:04.163475 7921 catalog_manager.cc:4841] TS e989b24be1854d28825b07473e75143f (127.0.98.2:34523): tablet 1817a1423478487ba3b98c192e68c2c5 (table test-table [id=fbc7186ac2184404b70ebb32b794fd04]) successfully deleted
I20260620 00:13:04.163506 8469 ts_tablet_manager.cc:1918] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:13:04.163566 8470 log.cc:1196] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/1817a1423478487ba3b98c192e68c2c5
I20260620 00:13:04.163592 8469 log.cc:1196] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/1817a1423478487ba3b98c192e68c2c5
I20260620 00:13:04.163821 8470 ts_tablet_manager.cc:1939] T 1817a1423478487ba3b98c192e68c2c5 P 09da830805e24056a7fe22ad92d3aef9: Deleting consensus metadata
I20260620 00:13:04.163846 8469 ts_tablet_manager.cc:1939] T 1817a1423478487ba3b98c192e68c2c5 P 00a45219c89249f69c9b31939d7b41ac: Deleting consensus metadata
I20260620 00:13:04.164206 7919 catalog_manager.cc:4841] TS 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715): tablet 1817a1423478487ba3b98c192e68c2c5 (table test-table [id=fbc7186ac2184404b70ebb32b794fd04]) successfully deleted
I20260620 00:13:04.164291 7919 catalog_manager.cc:4841] TS 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327): tablet 1817a1423478487ba3b98c192e68c2c5 (table test-table [id=fbc7186ac2184404b70ebb32b794fd04]) successfully deleted
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[7889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:13:04.188216 7983 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:04.182225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.0.98.62:44039)
0620 00:13:04.182299 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.182302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.182428 (+ 126us) server_negotiation.cc:407] Connection header received
0620 00:13:04.182517 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.182518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.182529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.182576 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:04.183161 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.183665 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.184500 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.184774 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.184860 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.188139 (+ 3279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56388: BlockingRecv error: recv got EOF from 127.0.0.1:56388 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[7889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:13:04.197781 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.189241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56394 (local address 127.0.98.62:44039)
0620 00:13:04.189294 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.189296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.189437 (+ 141us) server_negotiation.cc:407] Connection header received
0620 00:13:04.189513 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.189514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.189524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.189583 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:04.190156 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.190711 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.191479 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.191688 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.191771 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.194612 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:04.194615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:04.194615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:04.194640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:04.196334 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:04.196823 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:04.196825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:04.196825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:04.196886 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:04.197117 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:04.197118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:04.197118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:04.197234 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:04.197313 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.197495 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.197609 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:13:04.199959 7934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:13:04.200199 7934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:04.201872 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 6611523cab904c4b9590d8aedc768a63 (DEFAULT_TABLE table=test-table [id=330456fd84ea48c1b9d3b2881bd77a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.201923 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 6611523cab904c4b9590d8aedc768a63 (DEFAULT_TABLE table=test-table [id=330456fd84ea48c1b9d3b2881bd77a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.201915 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 6611523cab904c4b9590d8aedc768a63 (DEFAULT_TABLE table=test-table [id=330456fd84ea48c1b9d3b2881bd77a88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.202045 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6611523cab904c4b9590d8aedc768a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.202042 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6611523cab904c4b9590d8aedc768a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.202087 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6611523cab904c4b9590d8aedc768a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.203462 8446 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Bootstrap starting.
I20260620 00:13:04.203464 8447 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Bootstrap starting.
I20260620 00:13:04.203843 8447 tablet_bootstrap.cc:654] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.203909 8446 tablet_bootstrap.cc:654] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.204124 8445 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Bootstrap starting.
I20260620 00:13:04.204486 8445 tablet_bootstrap.cc:654] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.204802 8446 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: No bootstrap required, opened a new log
I20260620 00:13:04.204866 8446 ts_tablet_manager.cc:1397] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:04.205050 8446 raft_consensus.cc:348] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205142 8446 raft_consensus.cc:374] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.205163 8446 raft_consensus.cc:729] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a45219c89249f69c9b31939d7b41ac, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.205163 8445 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: No bootstrap required, opened a new log
I20260620 00:13:04.205230 8447 tablet_bootstrap.cc:492] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: No bootstrap required, opened a new log
I20260620 00:13:04.205226 8445 ts_tablet_manager.cc:1397] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:04.205209 8446 consensus_queue.cc:260] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205272 8447 ts_tablet_manager.cc:1397] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:13:04.205323 8446 ts_tablet_manager.cc:1428] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:04.205343 8445 raft_consensus.cc:348] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205412 8445 raft_consensus.cc:374] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.205433 8445 raft_consensus.cc:729] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.205402 8447 raft_consensus.cc:348] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205490 8447 raft_consensus.cc:374] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.205513 8447 raft_consensus.cc:729] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989b24be1854d28825b07473e75143f, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.205492 8445 consensus_queue.cc:260] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205554 8447 consensus_queue.cc:260] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.205581 8445 ts_tablet_manager.cc:1428] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:04.205636 8447 ts_tablet_manager.cc:1428] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:13:04.303041 8455 raft_consensus.cc:482] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:04.303166 8455 raft_consensus.cc:504] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.303360 8455 leader_election.cc:288] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), e989b24be1854d28825b07473e75143f (127.0.98.2:34523)
I20260620 00:13:04.303601 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6611523cab904c4b9590d8aedc768a63" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac" is_pre_election: true
I20260620 00:13:04.303611 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6611523cab904c4b9590d8aedc768a63" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f" is_pre_election: true
I20260620 00:13:04.303733 8091 raft_consensus.cc:2454] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:04.303730 8230 raft_consensus.cc:2454] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 0.
I20260620 00:13:04.303939 8303 leader_election.cc:302] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, 09da830805e24056a7fe22ad92d3aef9; no voters:
I20260620 00:13:04.304076 8455 raft_consensus.cc:2781] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:04.304121 8455 raft_consensus.cc:482] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:04.304154 8455 raft_consensus.cc:3037] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.304991 8455 raft_consensus.cc:504] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.305193 8455 leader_election.cc:288] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Requested vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), e989b24be1854d28825b07473e75143f (127.0.98.2:34523)
I20260620 00:13:04.305388 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6611523cab904c4b9590d8aedc768a63" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac"
I20260620 00:13:04.305462 8091 raft_consensus.cc:3037] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.305398 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6611523cab904c4b9590d8aedc768a63" candidate_uuid: "09da830805e24056a7fe22ad92d3aef9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e989b24be1854d28825b07473e75143f"
I20260620 00:13:04.305497 8230 raft_consensus.cc:3037] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.306138 8091 raft_consensus.cc:2454] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:04.306146 8230 raft_consensus.cc:2454] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09da830805e24056a7fe22ad92d3aef9 in term 1.
I20260620 00:13:04.306331 8303 leader_election.cc:302] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, 09da830805e24056a7fe22ad92d3aef9; no voters:
I20260620 00:13:04.306424 8455 raft_consensus.cc:2781] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:04.306480 8455 raft_consensus.cc:686] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Becoming Leader. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Running, Role: LEADER
I20260620 00:13:04.306551 8455 consensus_queue.cc:237] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.307211 7934 catalog_manager.cc:5495] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "09da830805e24056a7fe22ad92d3aef9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } health_report { overall_health: HEALTHY } } }
I20260620 00:13:04.327951 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.325097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.0.98.3:41327)
0620 00:13:04.325180 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.325182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.325191 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:04.325223 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.325224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.325235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.325296 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.325432 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.325961 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.327222 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.327411 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.327470 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.327618 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.327643 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.327678 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.327731 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.327748 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:13:04.328877 8091 raft_consensus.cc:1261] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.328905 8230 raft_consensus.cc:1261] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Refusing update from remote peer 09da830805e24056a7fe22ad92d3aef9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.329118 8455 consensus_queue.cc:1035] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.329224 8451 consensus_queue.cc:1035] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.331739 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.331837 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.332427 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Sending DeleteTablet for 3 replicas of tablet 6611523cab904c4b9590d8aedc768a63
I20260620 00:13:04.332752 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6611523cab904c4b9590d8aedc768a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41582
I20260620 00:13:04.332752 8211 tablet_service.cc:1514] Processing DeleteTablet for tablet 6611523cab904c4b9590d8aedc768a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42758
I20260620 00:13:04.332757 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 6611523cab904c4b9590d8aedc768a63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52390
I20260620 00:13:04.332851 8469 tablet_replica.cc:330] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: stopping tablet replica
I20260620 00:13:04.332851 8468 tablet_replica.cc:330] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: stopping tablet replica
I20260620 00:13:04.332871 8470 tablet_replica.cc:330] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: stopping tablet replica
I20260620 00:13:04.332897 8468 raft_consensus.cc:2229] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.332901 8469 raft_consensus.cc:2229] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.332918 8470 raft_consensus.cc:2229] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:13:04.332947 8468 raft_consensus.cc:2258] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.332952 8469 raft_consensus.cc:2258] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.332983 8470 raft_consensus.cc:2258] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.333277 8469 ts_tablet_manager.cc:1905] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.333341 8470 ts_tablet_manager.cc:1905] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.333326 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:13:04.333491 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:04.334685 8211 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a (DEFAULT_TABLE table=test-table [id=11007646aa9e4c5b876c1f7fb98229d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.334774 8071 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a (DEFAULT_TABLE table=test-table [id=11007646aa9e4c5b876c1f7fb98229d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.334827 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7e696aefa64606bacdf2038d25fe3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.334893 8071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7e696aefa64606bacdf2038d25fe3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.334998 8470 ts_tablet_manager.cc:1918] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:04.335047 8470 log.cc:1196] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/6611523cab904c4b9590d8aedc768a63
I20260620 00:13:04.335258 8470 ts_tablet_manager.cc:1939] T 6611523cab904c4b9590d8aedc768a63 P 09da830805e24056a7fe22ad92d3aef9: Deleting consensus metadata
I20260620 00:13:04.335943 8447 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: Bootstrap starting.
I20260620 00:13:04.336052 8468 ts_tablet_manager.cc:1905] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.336051 8446 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: Bootstrap starting.
I20260620 00:13:04.336290 8447 tablet_bootstrap.cc:654] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.336580 8446 tablet_bootstrap.cc:654] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.336944 8447 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: No bootstrap required, opened a new log
I20260620 00:13:04.336944 8348 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a (DEFAULT_TABLE table=test-table [id=11007646aa9e4c5b876c1f7fb98229d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:04.336985 8447 ts_tablet_manager.cc:1397] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:04.337045 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7e696aefa64606bacdf2038d25fe3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:04.337118 8447 raft_consensus.cc:348] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.337182 8447 raft_consensus.cc:374] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.337200 8447 raft_consensus.cc:729] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e989b24be1854d28825b07473e75143f, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.337256 8447 consensus_queue.cc:260] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.337348 8447 ts_tablet_manager.cc:1428] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:04.334794 8469 ts_tablet_manager.cc:1918] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:04.337488 8469 log.cc:1196] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/6611523cab904c4b9590d8aedc768a63
I20260620 00:13:04.337718 8469 ts_tablet_manager.cc:1939] T 6611523cab904c4b9590d8aedc768a63 P 00a45219c89249f69c9b31939d7b41ac: Deleting consensus metadata
I20260620 00:13:04.338130 8468 ts_tablet_manager.cc:1918] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:04.338204 8468 log.cc:1196] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/6611523cab904c4b9590d8aedc768a63
I20260620 00:13:04.338195 8445 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Bootstrap starting.
I20260620 00:13:04.338287 7919 catalog_manager.cc:4841] TS 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327): tablet 6611523cab904c4b9590d8aedc768a63 (table test-table [id=330456fd84ea48c1b9d3b2881bd77a88]) successfully deleted
I20260620 00:13:04.338389 7919 catalog_manager.cc:4841] TS 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715): tablet 6611523cab904c4b9590d8aedc768a63 (table test-table [id=330456fd84ea48c1b9d3b2881bd77a88]) successfully deleted
I20260620 00:13:04.338410 8468 ts_tablet_manager.cc:1939] T 6611523cab904c4b9590d8aedc768a63 P e989b24be1854d28825b07473e75143f: Deleting consensus metadata
I20260620 00:13:04.338521 8445 tablet_bootstrap.cc:654] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:04.338604 8446 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: No bootstrap required, opened a new log
I20260620 00:13:04.338644 8446 ts_tablet_manager.cc:1397] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:13:04.338732 7921 catalog_manager.cc:4841] TS e989b24be1854d28825b07473e75143f (127.0.98.2:34523): tablet 6611523cab904c4b9590d8aedc768a63 (table test-table [id=330456fd84ea48c1b9d3b2881bd77a88]) successfully deleted
I20260620 00:13:04.338765 8446 raft_consensus.cc:348] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.338825 8446 raft_consensus.cc:374] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.338858 8446 raft_consensus.cc:729] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a45219c89249f69c9b31939d7b41ac, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.338900 8446 consensus_queue.cc:260] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.338985 8446 ts_tablet_manager.cc:1428] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:04.339646 8445 tablet_bootstrap.cc:492] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: No bootstrap required, opened a new log
I20260620 00:13:04.339699 8445 ts_tablet_manager.cc:1397] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:13:04.339828 8445 raft_consensus.cc:348] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.339907 8445 raft_consensus.cc:374] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:04.339941 8445 raft_consensus.cc:729] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09da830805e24056a7fe22ad92d3aef9, State: Initialized, Role: FOLLOWER
I20260620 00:13:04.339988 8445 consensus_queue.cc:260] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.340071 8445 ts_tablet_manager.cc:1428] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:04.711906 8501 raft_consensus.cc:482] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:04.712019 8501 raft_consensus.cc:504] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.712173 8501 leader_election.cc:288] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327)
I20260620 00:13:04.712486 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7e696aefa64606bacdf2038d25fe3a" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac" is_pre_election: true
I20260620 00:13:04.712486 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7e696aefa64606bacdf2038d25fe3a" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da830805e24056a7fe22ad92d3aef9" is_pre_election: true
I20260620 00:13:04.712615 8091 raft_consensus.cc:2454] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 0.
I20260620 00:13:04.712618 8371 raft_consensus.cc:2454] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 0.
I20260620 00:13:04.712803 8163 leader_election.cc:302] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:04.712898 8501 raft_consensus.cc:2781] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:04.712941 8501 raft_consensus.cc:482] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:04.712960 8501 raft_consensus.cc:3037] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.713708 8501 raft_consensus.cc:504] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.713853 8501 leader_election.cc:288] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 election: Requested vote from peers 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715), 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327)
I20260620 00:13:04.714051 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7e696aefa64606bacdf2038d25fe3a" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09da830805e24056a7fe22ad92d3aef9"
I20260620 00:13:04.714067 8091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7e696aefa64606bacdf2038d25fe3a" candidate_uuid: "e989b24be1854d28825b07473e75143f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a45219c89249f69c9b31939d7b41ac"
I20260620 00:13:04.714144 8091 raft_consensus.cc:3037] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.714143 8371 raft_consensus.cc:3037] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:04.714906 8371 raft_consensus.cc:2454] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 1.
I20260620 00:13:04.714910 8091 raft_consensus.cc:2454] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e989b24be1854d28825b07473e75143f in term 1.
I20260620 00:13:04.715122 8163 leader_election.cc:302] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a45219c89249f69c9b31939d7b41ac, e989b24be1854d28825b07473e75143f; no voters:
I20260620 00:13:04.715252 8501 raft_consensus.cc:2781] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:04.715399 8501 raft_consensus.cc:686] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 1 LEADER]: Becoming Leader. State: Replica: e989b24be1854d28825b07473e75143f, State: Running, Role: LEADER
I20260620 00:13:04.715482 8501 consensus_queue.cc:237] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } }
I20260620 00:13:04.716144 7932 catalog_manager.cc:5495] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f reported cstate change: term changed from 0 to 1, leader changed from <none> to e989b24be1854d28825b07473e75143f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e989b24be1854d28825b07473e75143f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e989b24be1854d28825b07473e75143f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 } health_report { overall_health: UNKNOWN } } }
I20260620 00:13:04.786427 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.784051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.0.98.2:34523)
0620 00:13:04.784236 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.784238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.784245 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:13:04.784264 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.784265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.784275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.784342 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.784453 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.784967 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.785763 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.785970 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.786027 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.786176 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.786203 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.786236 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.786294 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.786312 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":136,"threads_started":1}
I20260620 00:13:04.791479 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.788944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49963 (local address 127.0.98.2:34523)
0620 00:13:04.788997 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:04.788999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:04.789101 (+ 102us) server_negotiation.cc:407] Connection header received
0620 00:13:04.789171 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:04.789172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:04.789181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:04.789228 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:04.789390 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.789845 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.790645 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.790881 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.790947 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.791131 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.791187 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.791252 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:04.791355 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:13:04.791393 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":32}
I20260620 00:13:04.791489 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.788896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.98.2:49963)
0620 00:13:04.789035 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.789047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:04.789106 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:04.789244 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:04.789245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:04.789247 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:13:04.789304 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.789306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.789918 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.789919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.790566 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:04.790568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:04.790983 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:04.790984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:04.791032 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:04.791037 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:13:04.791276 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:13:04.791278 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:13:04.791354 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:13:04.791409 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260620 00:13:04.793895 8371 raft_consensus.cc:1261] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Refusing update from remote peer e989b24be1854d28825b07473e75143f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.793896 8091 raft_consensus.cc:1261] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Refusing update from remote peer e989b24be1854d28825b07473e75143f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:04.794127 8528 consensus_queue.cc:1035] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a45219c89249f69c9b31939d7b41ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.794307 8526 consensus_queue.cc:1035] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09da830805e24056a7fe22ad92d3aef9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:04.805478 7934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.805599 7934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:04.806288 7934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e57984e283b4550bfdb90028d404d58: Sending DeleteTablet for 3 replicas of tablet 9b7e696aefa64606bacdf2038d25fe3a
I20260620 00:13:04.806574 8071 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41582
I20260620 00:13:04.806705 8469 tablet_replica.cc:330] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: stopping tablet replica
I20260620 00:13:04.806715 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52390
I20260620 00:13:04.806767 8469 raft_consensus.cc:2229] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.806800 8470 tablet_replica.cc:330] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: stopping tablet replica
I20260620 00:13:04.806813 8469 raft_consensus.cc:2258] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.806847 8470 raft_consensus.cc:2229] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:04.806896 8470 raft_consensus.cc:2258] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.807107 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b7e696aefa64606bacdf2038d25fe3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42758
I20260620 00:13:04.807202 8468 tablet_replica.cc:330] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: stopping tablet replica
I20260620 00:13:04.807193 8469 ts_tablet_manager.cc:1905] T 9b7e696aefa64606bacdf2038d25fe3a P 00a45219c89249f69c9b31939d7b41ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.807212 8470 ts_tablet_manager.cc:1905] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:04.807247 8468 raft_consensus.cc:2229] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:13:04.807341 8468 raft_consensus.cc:2258] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:04.807864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8006
I20260620 00:13:04.809068 8470 ts_tablet_manager.cc:1918] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:13:04.809150 8470 log.cc:1196] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/9b7e696aefa64606bacdf2038d25fe3a
I20260620 00:13:04.809369 8470 ts_tablet_manager.cc:1939] T 9b7e696aefa64606bacdf2038d25fe3a P 09da830805e24056a7fe22ad92d3aef9: Deleting consensus metadata
I20260620 00:13:04.809959 7919 catalog_manager.cc:4841] TS 09da830805e24056a7fe22ad92d3aef9 (127.0.98.3:41327): tablet 9b7e696aefa64606bacdf2038d25fe3a (table test-table [id=11007646aa9e4c5b876c1f7fb98229d7]) successfully deleted
I20260620 00:13:04.815825 8468 ts_tablet_manager.cc:1905] T 9b7e696aefa64606bacdf2038d25fe3a P e989b24be1854d28825b07473e75143f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:13:04.815865 7919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43715 (error 108)
I20260620 00:13:04.816095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8146
I20260620 00:13:04.818074 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.817170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43715 (local address 127.0.0.1:41588)
0620 00:13:04.817991 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.818033 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":54,"threads_started":1}
W20260620 00:13:04.818207 7919 catalog_manager.cc:4584] TS 00a45219c89249f69c9b31939d7b41ac (127.0.98.1:43715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b7e696aefa64606bacdf2038d25fe3a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43715: connect: Connection refused (error 111)
I20260620 00:13:04.822512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8286
I20260620 00:13:04.822537 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:04.822432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34523 (local address 127.0.0.1:42788)
0620 00:13:04.822474 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:04.822501 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 00:13:04.828518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 7902
2026-06-20T00:13:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3043 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[8541](info): set up 2 sockets
Jun 20 00:13:04 dist-test-slave-28bv krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260620 00:13:05.872978 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:13:05 dist-test-slave-28bv krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:13:05.999470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43599
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:06.081586 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:06.081836 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:06.081861 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:06.083190 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:06.083266 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:06.083282 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:06.083297 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:06.083309 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:06.085503 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:06.085748 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:06.085938 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:06.091372 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:06.091483 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:06.091390 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:06.091470 8554 server_base.cc:1034] running on GCE node
I20260620 00:13:06.092047 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:06.092437 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:06.093626 8554 hybrid_clock.cc:648] HybridClock initialized: now 1781914386093603 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T00:13:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1267 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:13:06 dist-test-slave-28bv krb5kdc[8567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:13:06 dist-test-slave-28bv krb5kdc[8567](info): set up 2 sockets
Jun 20 00:13:06 dist-test-slave-28bv krb5kdc[8567](info): commencing operation
krb5kdc: starting...
W20260620 00:13:07.145112 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:13:07.268702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:07.352099 8580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:07.352324 8580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:07.352347 8580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:07.353627 8580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:07.353677 8580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:07.353691 8580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:07.353704 8580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:07.353716 8580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:07.355213 8580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:07.355413 8580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:07.355594 8580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:07.360513 8585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.360518 8588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.360517 8586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.360533 8580 server_base.cc:1034] running on GCE node
I20260620 00:13:07.361068 8580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:07.361367 8580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:07.362522 8580 hybrid_clock.cc:648] HybridClock initialized: now 1781914387362512 us; error 55 us; skew 500 ppm
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:07.370836 8580 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:13:07.371407 8580 webserver.cc:460] Webserver started at http://127.0.98.62:33977/ using document root <none> and password file <none>
I20260620 00:13:07.371668 8580 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:07.371717 8580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:07.371867 8580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:07.373064 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "ec547518359848f2b44e005a8423a055"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "d143e66fb6672b1ed99475a6eb7583eb"
server_key_iv: "dd00a33e5352a44805866f2609ae1f12"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.373437 8580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec547518359848f2b44e005a8423a055"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "d143e66fb6672b1ed99475a6eb7583eb"
server_key_iv: "dd00a33e5352a44805866f2609ae1f12"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.374854 8580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:13:07.375694 8595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.375877 8580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:07.375972 8580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "ec547518359848f2b44e005a8423a055"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "d143e66fb6672b1ed99475a6eb7583eb"
server_key_iv: "dd00a33e5352a44805866f2609ae1f12"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.376071 8580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:07.389309 8580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:07.391618 8580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:07.391891 8580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:07.396127 8580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45767
I20260620 00:13:07.396162 8647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45767 every 8 connection(s)
I20260620 00:13:07.396715 8580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:07.397434 8648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:07.399808 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Bootstrap starting.
I20260620 00:13:07.400434 8648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:07.400766 8648 log.cc:824] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:07.401479 8648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: No bootstrap required, opened a new log
I20260620 00:13:07.403116 8648 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:07.403270 8648 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:07.403291 8648 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec547518359848f2b44e005a8423a055, State: Initialized, Role: FOLLOWER
I20260620 00:13:07.403447 8648 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:07.403518 8648 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:07.403541 8648 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:07.403599 8648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:07.403823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8580
I20260620 00:13:07.403898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:13:07.404078 392 external_mini_cluster.cc:1428] Setting key fb69cc459c4d0134f3be5f8cc15fa9c1
I20260620 00:13:07.404798 8648 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:07.404909 8648 leader_election.cc:302] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec547518359848f2b44e005a8423a055; no voters:
I20260620 00:13:07.405061 8648 leader_election.cc:288] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:13:07.405092 8653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:07.405274 8653 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 LEADER]: Becoming Leader. State: Replica: ec547518359848f2b44e005a8423a055, State: Running, Role: LEADER
I20260620 00:13:07.405308 8648 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:07.405464 8653 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:07.405992 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec547518359848f2b44e005a8423a055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } } }
I20260620 00:13:07.406118 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:07.405997 8655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec547518359848f2b44e005a8423a055. Latest consensus state: current_term: 1 leader_uuid: "ec547518359848f2b44e005a8423a055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } } }
I20260620 00:13:07.406193 8655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:07.406399 8662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:07.406934 8662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:07.408591 8662 catalog_manager.cc:1347] Generated new cluster ID: bf1c8f226e7d41d6858e9335de54a1fa
I20260620 00:13:07.408643 8662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:07.415987 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.404620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.98.62:45767)
0620 00:13:07.404843 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:07.404853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:07.404897 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:13:07.405035 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:07.405043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:07.405106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:07.405266 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:07.406055 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.407451 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.408393 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.409040 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.409165 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.412009 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:07.412014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:07.412014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:07.412033 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:07.414166 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.414615 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.414618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.414618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.414736 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.414959 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.414960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.414960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.415283 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:07.415559 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:07.415628 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:13:07.415766 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":114,"threads_started":1}
I20260620 00:13:07.420228 8662 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:13:07.420889 8662 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:07.431406 8662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Generated new TSK 0
I20260620 00:13:07.431742 8662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:13:07.468760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:07.568289 8674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:07.568601 8674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:07.568670 8674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:07.569909 8674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:07.569975 8674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:07.570025 8674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:07.571647 8674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:07.571923 8674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:07.572160 8674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:13:07.577450 8674 server_base.cc:1034] running on GCE node
W20260620 00:13:07.577431 8682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.577426 8679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.577414 8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.577916 8674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:07.578156 8674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:07.579363 8674 hybrid_clock.cc:648] HybridClock initialized: now 1781914387579339 us; error 41 us; skew 500 ppm
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:07.582137 8674 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:13:07.582685 8674 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260620 00:13:07.582934 8674 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:07.583006 8674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:07.583184 8674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:07.584416 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "29478692bc45494893c55738658a4e40"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "5a6d69060abd5e2fb58cc6d1c6e7dc5c"
server_key_iv: "91737f6a28f2a2ce22d836d5bb1565e3"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.584964 8674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29478692bc45494893c55738658a4e40"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "5a6d69060abd5e2fb58cc6d1c6e7dc5c"
server_key_iv: "91737f6a28f2a2ce22d836d5bb1565e3"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.586587 8674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:13:07.587332 8689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.587522 8674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:07.587616 8674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "29478692bc45494893c55738658a4e40"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "5a6d69060abd5e2fb58cc6d1c6e7dc5c"
server_key_iv: "91737f6a28f2a2ce22d836d5bb1565e3"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.587694 8674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:07.599143 8674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:07.601455 8674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:07.601646 8674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:07.601894 8674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:07.602226 8674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:07.602279 8674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.602319 8674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:07.602345 8674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.608657 8674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39779
I20260620 00:13:07.608709 8802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39779 every 8 connection(s)
I20260620 00:13:07.609023 8674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:13:07.614768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8674
I20260620 00:13:07.614940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:13:07.615166 392 external_mini_cluster.cc:1428] Setting key 7047432c209774059fa6ecfbeccdf676
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:07.620246 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.609755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53217 (local address 127.0.98.62:45767)
0620 00:13:07.609824 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:07.609826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:07.610020 (+ 194us) server_negotiation.cc:407] Connection header received
0620 00:13:07.611022 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:07.611023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:07.611038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:07.611091 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:07.611704 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.612198 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.613430 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.613706 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.613785 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.616251 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:07.616253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:07.616253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:07.616264 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:07.618262 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.618853 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.618855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.618855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.618926 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.619234 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.619235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.619236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.619411 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:07.619494 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:07.619922 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 00:13:07.620081 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:13:07.620361 8805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.609762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:53217)
0620 00:13:07.609914 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:07.609950 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:07.610909 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:07.611148 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:07.611152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:07.611415 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:07.611615 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.611619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.612311 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.612313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.613315 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.613318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.613798 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.613799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.613903 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.614423 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:07.614433 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:07.616124 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:07.618360 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.618368 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.618371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.618739 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.619004 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.619005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.619005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.619146 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.619483 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:07.619496 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:07.619726 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 00:13:07.619855 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 00:13:07.620050 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260620 00:13:07.620903 8803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45767
I20260620 00:13:07.621013 8803 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:07.621246 8803 heartbeater.cc:511] Master 127.0.98.62:45767 requested a full tablet report, sending...
I20260620 00:13:07.621800 8612 ts_manager.cc:194] Registered new tserver with Master: 29478692bc45494893c55738658a4e40 (127.0.98.1:39779)
I20260620 00:13:07.622424 8612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53217
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:13:07.645215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:07.727895 8808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:07.728121 8808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:07.728147 8808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:07.729388 8808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:07.729435 8808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:07.729462 8808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:07.731014 8808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:07.731272 8808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:07.731523 8808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:07.736631 8813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.736651 8816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.736673 8808 server_base.cc:1034] running on GCE node
W20260620 00:13:07.736640 8814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.737041 8808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:07.737304 8808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:07.738473 8808 hybrid_clock.cc:648] HybridClock initialized: now 1781914387738452 us; error 33 us; skew 500 ppm
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:07.741348 8808 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:13:07.741849 8808 webserver.cc:460] Webserver started at http://127.0.98.2:35309/ using document root <none> and password file <none>
I20260620 00:13:07.742085 8808 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:07.742132 8808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:07.742257 8808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:07.743321 8808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "847d9a75a8824e85b2421db4d706d8d4"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "438478e346534900e81f76d2032d8dc6"
server_key_iv: "1fa184bae792079d6c24a7e94bde5629"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.743680 8808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "847d9a75a8824e85b2421db4d706d8d4"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "438478e346534900e81f76d2032d8dc6"
server_key_iv: "1fa184bae792079d6c24a7e94bde5629"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.745092 8808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:13:07.745918 8823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.746085 8808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:13:07.746173 8808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "847d9a75a8824e85b2421db4d706d8d4"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "438478e346534900e81f76d2032d8dc6"
server_key_iv: "1fa184bae792079d6c24a7e94bde5629"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.746233 8808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:07.752732 8808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:07.754982 8808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:07.755095 8808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:07.755283 8808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:07.755580 8808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:07.755613 8808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.755653 8808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:07.755687 8808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.762064 8808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38053
I20260620 00:13:07.762142 8936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38053 every 8 connection(s)
I20260620 00:13:07.762457 8808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:07.770296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8808
I20260620 00:13:07.770407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:13:07.770617 392 external_mini_cluster.cc:1428] Setting key 69ae52c96c79632ac2355cf82907a7ec
I20260620 00:13:07.772701 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.763253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56647 (local address 127.0.98.62:45767)
0620 00:13:07.763319 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:07.763321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:07.763516 (+ 195us) server_negotiation.cc:407] Connection header received
0620 00:13:07.764490 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:07.764491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:07.764502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:07.764571 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:07.765243 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.765735 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.766724 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.766928 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.766990 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.769409 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:07.769411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:07.769411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:07.769419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:07.770987 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.771448 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.771449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.771449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.771505 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.771852 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.771853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.771854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.772040 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:07.772107 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:07.772477 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 00:13:07.772624 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:13:07.773061 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.763226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:56647)
0620 00:13:07.763393 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:07.763430 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:07.764389 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:07.764669 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:07.764672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:07.764936 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:07.765144 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.765148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.765821 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.765822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.766647 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.766649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.767035 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.767036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.767158 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.767740 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:07.767749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:07.769310 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:07.771080 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.771088 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.771090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.771372 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.771571 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.771572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.771572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.771743 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.772115 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:07.772121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:07.772306 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 00:13:07.772440 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 00:13:07.772667 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260620 00:13:07.773542 8937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45767
I20260620 00:13:07.773671 8937 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:07.773857 8937 heartbeater.cc:511] Master 127.0.98.62:45767 requested a full tablet report, sending...
I20260620 00:13:07.774242 8612 ts_manager.cc:194] Registered new tserver with Master: 847d9a75a8824e85b2421db4d706d8d4 (127.0.98.2:38053)
I20260620 00:13:07.774792 8612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56647
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:13:07.799144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:07.882491 8942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:07.882721 8942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:07.882747 8942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:07.884054 8942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:07.884102 8942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:07.884131 8942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:07.885779 8942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:07.886053 8942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:07.886304 8942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:07.892092 8947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.892167 8942 server_base.cc:1034] running on GCE node
W20260620 00:13:07.892092 8948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:07.892092 8950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:07.892606 8942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:07.892915 8942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:07.894122 8942 hybrid_clock.cc:648] HybridClock initialized: now 1781914387894114 us; error 38 us; skew 500 ppm
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:07.896829 8942 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:13:07.897327 8942 webserver.cc:460] Webserver started at http://127.0.98.3:38281/ using document root <none> and password file <none>
I20260620 00:13:07.897564 8942 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:07.897612 8942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:07.897734 8942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:07.898799 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3c75b3cddab435f91d60dd39ec70bce"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "ef62a28562685852306fb17f99d38983"
server_key_iv: "ecb3b56192c5e2882930ce369538bbd0"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.899151 8942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3c75b3cddab435f91d60dd39ec70bce"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "ef62a28562685852306fb17f99d38983"
server_key_iv: "ecb3b56192c5e2882930ce369538bbd0"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.900532 8942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 00:13:07.901360 8957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.901523 8942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:13:07.901620 8942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "a3c75b3cddab435f91d60dd39ec70bce"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "ef62a28562685852306fb17f99d38983"
server_key_iv: "ecb3b56192c5e2882930ce369538bbd0"
server_key_version: "encryptionkey@0"
I20260620 00:13:07.901696 8942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:07.924901 8942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:07.927157 8942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:07.927327 8942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:07.927572 8942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:07.927896 8942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:07.927945 8942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.927986 8942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:07.928014 8942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:07.934121 8942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46027
I20260620 00:13:07.934170 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46027 every 8 connection(s)
I20260620 00:13:07.934532 8942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:13:07 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:07.943989 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.935303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49693 (local address 127.0.98.62:45767)
0620 00:13:07.935372 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:07.935375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:07.935583 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:13:07.936619 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:07.936620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:07.936634 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:07.936703 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:07.937342 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.937867 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.938877 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.939060 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.939118 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.941377 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:07.941378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:07.941378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:07.941385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:07.942774 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.943160 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.943160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.943161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.943211 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:07.943468 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:07.943469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:07.943469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:07.943570 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:07.943619 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:07.943855 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 00:13:07.943920 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:13:07.944283 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:07.935293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:49693)
0620 00:13:07.935464 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:07.935500 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:07.936430 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:07.936787 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:07.936790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:07.937053 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:07.937251 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.937255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.937951 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.937952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.938794 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:07.938797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:07.939131 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:07.939131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:07.939245 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:07.939758 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:07.939767 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:07.941281 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:07.942853 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.942860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.942862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.943097 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.943287 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:07.943288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:07.943288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:07.943405 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:07.943624 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:07.943628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:07.943747 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 00:13:07.943831 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:13:07.943968 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260620 00:13:07.944630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 8942
I20260620 00:13:07.944738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:13:07.944953 392 external_mini_cluster.cc:1428] Setting key c54888af484272781a459b55b3f9a3a9
I20260620 00:13:07.944959 9071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45767
I20260620 00:13:07.945067 9071 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:07.945266 9071 heartbeater.cc:511] Master 127.0.98.62:45767 requested a full tablet report, sending...
I20260620 00:13:07.945761 8611 ts_manager.cc:194] Registered new tserver with Master: a3c75b3cddab435f91d60dd39ec70bce (127.0.98.3:46027)
I20260620 00:13:07.946146 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49693
I20260620 00:13:07.957031 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:13:08.036726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8674
I20260620 00:13:08.041326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8808
I20260620 00:13:08.046132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8942
I20260620 00:13:08.050657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 8580
I20260620 00:13:08.056005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45767
--webserver_interface=127.0.98.62
--webserver_port=33977
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:08.140199 9080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:08.140429 9080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:08.140453 9080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:08.141711 9080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:08.141762 9080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:08.141777 9080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:08.141788 9080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:08.141800 9080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:08.143396 9080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:08.143674 9080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:08.143913 9080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:08.148773 9085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.148785 9088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.148773 9086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:08.149142 9080 server_base.cc:1034] running on GCE node
I20260620 00:13:08.149389 9080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:08.149693 9080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:08.150842 9080 hybrid_clock.cc:648] HybridClock initialized: now 1781914388150824 us; error 34 us; skew 500 ppm
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:08.153311 9080 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:13:08.153851 9080 webserver.cc:460] Webserver started at http://127.0.98.62:33977/ using document root <none> and password file <none>
I20260620 00:13:08.154079 9080 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:08.154176 9080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:08.155568 9080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:08.156293 9095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.156473 9080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:08.156598 9080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "ec547518359848f2b44e005a8423a055"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "d143e66fb6672b1ed99475a6eb7583eb"
server_key_iv: "dd00a33e5352a44805866f2609ae1f12"
server_key_version: "encryptionkey@0"
I20260620 00:13:08.156988 9080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:08.187891 9080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:08.189975 9080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:08.190141 9080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:08.194310 9080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45767
I20260620 00:13:08.194350 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45767 every 8 connection(s)
I20260620 00:13:08.194921 9080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:08.196221 9148 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:13:08.196933 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Bootstrap starting.
I20260620 00:13:08.201581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9080
I20260620 00:13:08.201916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39779
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45767
--webserver_port=45649
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:13:08.205264 9148 log.cc:824] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:08.206982 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:13:08.207280 9148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Bootstrap complete.
I20260620 00:13:08.209671 9148 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:08.209929 9148 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec547518359848f2b44e005a8423a055, State: Initialized, Role: FOLLOWER
I20260620 00:13:08.210131 9148 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:08.210225 9148 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:08.210279 9148 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:08.210328 9148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:13:08.211359 9148 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:08.211479 9148 leader_election.cc:302] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec547518359848f2b44e005a8423a055; no voters:
I20260620 00:13:08.211665 9148 leader_election.cc:288] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:13:08.211735 9152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:13:08.211938 9148 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:08.211953 9152 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [term 2 LEADER]: Becoming Leader. State: Replica: ec547518359848f2b44e005a8423a055, State: Running, Role: LEADER
I20260620 00:13:08.212236 9152 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } }
I20260620 00:13:08.212518 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec547518359848f2b44e005a8423a055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } } }
I20260620 00:13:08.212618 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:08.212888 9154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec547518359848f2b44e005a8423a055. Latest consensus state: current_term: 2 leader_uuid: "ec547518359848f2b44e005a8423a055" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec547518359848f2b44e005a8423a055" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45767 } } }
I20260620 00:13:08.212947 9154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:08.213177 9159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:08.213671 9159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:08.213847 9159 catalog_manager.cc:1259] Loaded cluster ID: bf1c8f226e7d41d6858e9335de54a1fa
I20260620 00:13:08.214026 9159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:13:08.215306 9159 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:08.215480 9159 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec547518359848f2b44e005a8423a055: Loaded TSK: 0
I20260620 00:13:08.215690 9159 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:13:08.286111 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:08.286325 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:08.286350 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:08.287582 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:08.287642 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:08.287672 9150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:08.289188 9150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45649
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:08.289448 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:08.289700 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:08.294961 9175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.295113 9176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.294965 9178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:08.295396 9150 server_base.cc:1034] running on GCE node
I20260620 00:13:08.295637 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:08.295914 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:08.297117 9150 hybrid_clock.cc:648] HybridClock initialized: now 1781914388297098 us; error 34 us; skew 500 ppm
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914388, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:08.300041 9150 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:13:08.300625 9150 webserver.cc:460] Webserver started at http://127.0.98.1:45649/ using document root <none> and password file <none>
I20260620 00:13:08.300906 9150 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:08.300987 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:08.302757 9150 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:13:08.303500 9185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.303778 9150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:08.303879 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "29478692bc45494893c55738658a4e40"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "5a6d69060abd5e2fb58cc6d1c6e7dc5c"
server_key_iv: "91737f6a28f2a2ce22d836d5bb1565e3"
server_key_version: "encryptionkey@0"
I20260620 00:13:08.304317 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:08.327397 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:08.330138 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:08.330305 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:08.330569 9150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:08.330942 9150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:08.330991 9150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.331022 9150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:08.331046 9150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.338209 9150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39779
I20260620 00:13:08.338250 9298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39779 every 8 connection(s)
I20260620 00:13:08.338706 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.347384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9150
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.347721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45767
--webserver_port=35309
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:13:08.348662 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.339486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:56595)
0620 00:13:08.339658 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.339729 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.340248 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.340761 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.340764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.341365 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.341558 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.341562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.343237 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.343237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.344041 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.344043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.344856 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.344856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.344963 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.345537 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.345545 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.348381 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260620 00:13:08.348730 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.339597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56595 (local address 127.0.98.62:45767)
0620 00:13:08.339799 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.339808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.339850 (+ 42us) server_negotiation.cc:407] Connection header received
0620 00:13:08.340465 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.340473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.340531 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.340692 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.341681 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.343132 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.344130 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.344765 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.344882 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.348520 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56595: BlockingRecv error: recv got EOF from 127.0.98.1:56595 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
W20260620 00:13:08.349290 9299 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45767 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45767: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.355563 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.349598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:54491)
0620 00:13:08.349749 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.349758 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.349833 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.350018 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.350019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.350165 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.350213 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.350215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.351164 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.351165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.352049 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.352051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.352662 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.352663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.352730 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.353330 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.353335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.355482 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260620 00:13:08.355679 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.349633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54491 (local address 127.0.98.62:45767)
0620 00:13:08.349677 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.349679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.349817 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:13:08.349911 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.349912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.349926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.349972 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.350293 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.351062 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.352144 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.352503 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.352631 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.355596 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54491: BlockingRecv error: recv got EOF from 127.0.98.1:54491 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.361295 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.356071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:46723)
0620 00:13:08.356168 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.356179 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.356223 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.356370 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.356371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.356512 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.356585 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.356586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.357523 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.357524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.358102 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.358104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.358615 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.358616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.358675 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.359176 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.359181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.361234 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 00:13:08.361409 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.356128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46723 (local address 127.0.98.62:45767)
0620 00:13:08.356191 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.356194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.356208 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:13:08.356277 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.356278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.356292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.356357 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.356650 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.357420 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.358192 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.358520 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.358600 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.361342 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46723: BlockingRecv error: recv got EOF from 127.0.98.1:46723 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 00:13:08.361441 9299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:13:08.434262 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:08.434525 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:08.434566 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:08.435833 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:08.435914 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:08.435961 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:08.437544 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35309
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:08.437866 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:08.438114 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:08.443962 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.443989 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.443969 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:08.444205 9303 server_base.cc:1034] running on GCE node
I20260620 00:13:08.444501 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:08.444810 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:08.445999 9303 hybrid_clock.cc:648] HybridClock initialized: now 1781914388445970 us; error 46 us; skew 500 ppm
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914388, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:08.448885 9303 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:13:08.449386 9303 webserver.cc:460] Webserver started at http://127.0.98.2:35309/ using document root <none> and password file <none>
I20260620 00:13:08.449636 9303 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:08.449723 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:08.451181 9303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:08.451946 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.452149 9303 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:08.452265 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "847d9a75a8824e85b2421db4d706d8d4"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "438478e346534900e81f76d2032d8dc6"
server_key_iv: "1fa184bae792079d6c24a7e94bde5629"
server_key_version: "encryptionkey@0"
I20260620 00:13:08.452757 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:08.472973 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:08.475325 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:08.475499 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:08.475797 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:08.476142 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:08.476200 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.476243 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:08.476282 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.482339 9303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38053
I20260620 00:13:08.482381 9432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38053 every 8 connection(s)
I20260620 00:13:08.482750 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:13:08.483829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9303
I20260620 00:13:08.484122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46027
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45767
--webserver_port=38281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:13:08.492516 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.483470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55805 (local address 127.0.98.62:45767)
0620 00:13:08.483538 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.483540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.483800 (+ 260us) server_negotiation.cc:407] Connection header received
0620 00:13:08.484443 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.484445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.484464 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.484580 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.486072 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.486644 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.488060 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.488302 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.488400 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.492459 (+ 4059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55805: BlockingRecv error: recv got EOF from 127.0.98.2:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:13:08.492650 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.483434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:55805)
0620 00:13:08.483601 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.483700 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.484326 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.484699 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.484703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.485497 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.485910 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.485919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.486899 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.486900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.487941 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.487944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.488412 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.488414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.488617 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.489273 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.489291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.492346 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260620 00:13:08.492908 9433 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45767 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45767: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.497644 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.493189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:56701)
0620 00:13:08.493226 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.493238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.493282 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.493411 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.493412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.493570 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.493624 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.493625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.494271 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.494272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.494779 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.494780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.495158 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.495159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.495227 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.495752 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.495757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.497552 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 00:13:08.497704 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.493235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56701 (local address 127.0.98.62:45767)
0620 00:13:08.493290 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.493292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.493304 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:08.493344 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.493345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.493355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.493404 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.493699 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.494194 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.494862 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.495067 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.495166 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.497636 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56701: BlockingRecv error: recv got EOF from 127.0.98.2:56701 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.503183 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.497922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:53419)
0620 00:13:08.498082 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.498095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.498222 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.498372 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.498372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.498521 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.498560 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.498561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.499210 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.499211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.500134 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.500137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.500514 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.500515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.500608 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.501071 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.501078 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.503106 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139}
W20260620 00:13:08.503419 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.498031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53419 (local address 127.0.98.62:45767)
0620 00:13:08.498095 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.498097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.498209 (+ 112us) server_negotiation.cc:407] Connection header received
0620 00:13:08.498288 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.498289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.498299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.498354 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.498626 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.499141 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.500234 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.500413 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.500477 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.503215 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53419: BlockingRecv error: recv got EOF from 127.0.98.2:53419 (error 108)
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":45}
W20260620 00:13:08.503506 9433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:13:08.571914 9436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:08.572136 9436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:08.572160 9436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:08.573428 9436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:08.573479 9436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:08.573508 9436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:08.574978 9436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38281
--tserver_master_addrs=127.0.98.62:45767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:08.575233 9436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:08.575462 9436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:08.580862 9444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:08.580870 9442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:08.580906 9436 server_base.cc:1034] running on GCE node
W20260620 00:13:08.580869 9441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:08.581291 9436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:08.581542 9436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:08.582707 9436 hybrid_clock.cc:648] HybridClock initialized: now 1781914388582691 us; error 32 us; skew 500 ppm
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914388, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:08.585242 9436 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:13:08.585747 9436 webserver.cc:460] Webserver started at http://127.0.98.3:38281/ using document root <none> and password file <none>
I20260620 00:13:08.585990 9436 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:08.586071 9436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:08.587419 9436 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:08.588057 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.588243 9436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:08.588347 9436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "a3c75b3cddab435f91d60dd39ec70bce"
format_stamp: "Formatted at 2026-06-20 00:13:07 on dist-test-slave-28bv"
server_key: "ef62a28562685852306fb17f99d38983"
server_key_iv: "ecb3b56192c5e2882930ce369538bbd0"
server_key_version: "encryptionkey@0"
I20260620 00:13:08.588749 9436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:08.605537 9436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:08.607570 9436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:08.607748 9436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:08.607992 9436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:08.608359 9436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:08.608412 9436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.608445 9436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:08.608474 9436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:08.615329 9436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46027
I20260620 00:13:08.615371 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46027 every 8 connection(s)
I20260620 00:13:08.615724 9436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:13:08.619952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9436
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:13:08.624419 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.616445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56111 (local address 127.0.98.62:45767)
0620 00:13:08.616494 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.616496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.616747 (+ 251us) server_negotiation.cc:407] Connection header received
0620 00:13:08.617243 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.617244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.617254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.617322 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.618376 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.618990 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.620248 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.620587 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.620658 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.624369 (+ 3711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56111: BlockingRecv error: recv got EOF from 127.0.98.3:56111 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:13:08.624521 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.616424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:56111)
0620 00:13:08.616611 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.616655 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.617136 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.617394 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.617397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.617991 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.618213 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.618217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.619090 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.619091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.620145 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.620148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.620733 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.620734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.620845 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.621489 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.621500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.624313 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
W20260620 00:13:08.624773 9565 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45767 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45767: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.629344 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.621028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.0.98.62:45767)
0620 00:13:08.621670 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.621672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.621681 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:08.621731 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.621732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.621745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.621812 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.622687 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.623443 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.624288 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.624599 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.624668 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.626028 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:08.626034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:08.626034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:08.626046 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:08.627799 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:08.628172 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:08.628175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:08.628175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:08.628268 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:08.628484 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:08.628485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:08.628485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:08.628809 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:08.628959 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:08.629095 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 00:13:08.629208 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":99,"threads_started":1}
I20260620 00:13:08.629372 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.625008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:39605)
0620 00:13:08.625056 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.625066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.625102 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.625501 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.625502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.625628 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.625672 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.625674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.626277 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.626278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.626792 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.626793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.627149 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.627150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.627212 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.627545 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.627549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.629294 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:13:08.629411 9302 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.625346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39605 (local address 127.0.98.62:45767)
0620 00:13:08.625393 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.625395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.625402 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:13:08.625421 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.625421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.625430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.625495 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.625761 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.626205 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.626872 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.627070 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.627147 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.629355 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39605: BlockingRecv error: recv got EOF from 127.0.98.3:39605 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:08 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:08.633498 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:08.629668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:43045)
0620 00:13:08.629707 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:08.629718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:08.629759 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:08.629883 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:08.629884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:08.630017 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:08.630048 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.630049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.630659 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.630660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.631150 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:08.631151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.631524 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.631525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:08.631577 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.631918 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:08.631921 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:08.633448 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 00:13:08.633680 9565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:13:08.633764 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:08.629749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43045 (local address 127.0.98.62:45767)
0620 00:13:08.629782 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:08.629784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:08.629791 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:13:08.629810 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:08.629810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:08.629820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:08.629878 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:08.630128 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.630583 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.631242 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:08.631447 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:08.631523 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:08.633646 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43045: BlockingRecv error: recv got EOF from 127.0.98.3:43045 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:09.366986 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:09.361869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:34447)
0620 00:13:09.362024 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:09.362035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:09.362086 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:09.362318 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:09.362319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:09.362501 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:09.362573 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.362575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.363313 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.363314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.363914 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.363917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.364314 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.364315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.364369 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.364939 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:09.364944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:09.366920 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
W20260620 00:13:09.367193 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:09.361970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34447 (local address 127.0.98.62:45767)
0620 00:13:09.362086 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:09.362088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:09.362128 (+ 40us) server_negotiation.cc:407] Connection header received
0620 00:13:09.362180 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:09.362181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:09.362194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:09.362260 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:09.362673 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.363214 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.364031 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.364227 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.364300 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.367119 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34447: BlockingRecv error: recv got EOF from 127.0.98.1:34447 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:09.509199 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:09.504063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:54799)
0620 00:13:09.504264 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:09.504280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:09.504349 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:09.504510 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:09.504511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:09.504705 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:09.504759 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.504761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.505438 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.505439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.506010 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.506012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.506421 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.506422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.506478 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.507025 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:09.507031 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:09.509128 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
W20260620 00:13:09.509368 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:09.504228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54799 (local address 127.0.98.62:45767)
0620 00:13:09.504313 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:09.504315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:09.504332 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:13:09.504432 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:09.504433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:09.504447 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:09.504494 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:09.504844 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.505355 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.506110 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.506359 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.506432 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.509271 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54799: BlockingRecv error: recv got EOF from 127.0.98.2:54799 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:09 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:09.639173 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:09.634119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:35807)
0620 00:13:09.634277 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:09.634295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:09.634368 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:09.634545 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:09.634545 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:09.634710 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:09.634766 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.634768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.635506 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.635507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.636124 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:09.636127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.636513 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.636514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:09.636638 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.637175 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:09.637180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:09.639097 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
W20260620 00:13:09.639321 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:09.634204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35807 (local address 127.0.98.62:45767)
0620 00:13:09.634261 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:09.634268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:09.634428 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:13:09.634459 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:09.634460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:09.634474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:09.634522 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:09.634861 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.635408 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.636215 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:09.636424 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:09.636486 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:09.639206 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35807: BlockingRecv error: recv got EOF from 127.0.98.3:35807 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:10.372783 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:10.367472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:36003)
0620 00:13:10.367619 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:10.367634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:10.367711 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:10.367867 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:10.367868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:10.368076 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:10.368151 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.368154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.369088 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.369089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.369643 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.369645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.370028 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.370029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.370072 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.370664 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:10.370669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:10.372683 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
W20260620 00:13:10.372843 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:10.367584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36003 (local address 127.0.98.62:45767)
0620 00:13:10.367689 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:10.367691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:10.367707 (+ 16us) server_negotiation.cc:407] Connection header received
0620 00:13:10.367787 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:10.367788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:10.367798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:10.367857 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:10.368235 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.369003 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.369752 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.369959 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.370018 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.372765 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36003: BlockingRecv error: recv got EOF from 127.0.98.1:36003 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:10.515568 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:10.509889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:48029)
0620 00:13:10.510081 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:10.510100 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:10.510165 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:10.510366 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:10.510368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:10.510536 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:10.510608 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.510610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.511373 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.511374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.512050 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.512053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.512426 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.512427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.512477 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.513319 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:10.513324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:10.515465 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
W20260620 00:13:10.515623 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:10.510018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48029 (local address 127.0.98.62:45767)
0620 00:13:10.510083 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:10.510085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:10.510162 (+ 77us) server_negotiation.cc:407] Connection header received
0620 00:13:10.510222 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:10.510222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:10.510234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:10.510291 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:10.510708 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.511251 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.512137 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.512350 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.512408 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.515557 (+ 3149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48029: BlockingRecv error: recv got EOF from 127.0.98.2:48029 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:10 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:10.644800 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:10.639767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:53529)
0620 00:13:10.640034 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:10.640068 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:10.640139 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:10.640279 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:10.640280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:10.640503 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:10.640573 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.640575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.641295 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.641296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.641823 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:10.641825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.642267 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.642268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:10.642314 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.642803 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:10.642821 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:10.644735 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":116,"threads_started":1}
W20260620 00:13:10.644927 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:10.639905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53529 (local address 127.0.98.62:45767)
0620 00:13:10.639964 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:10.639967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:10.640153 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:13:10.640186 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:10.640187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:10.640204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:10.640260 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:10.640650 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.641196 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.641989 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:10.642194 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:10.642257 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:10.644825 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53529: BlockingRecv error: recv got EOF from 127.0.98.3:53529 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:11.378120 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:11.373471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:33583)
0620 00:13:11.373614 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:11.373624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:11.373674 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:11.373807 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:11.373808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:11.374011 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:11.374059 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.374060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.374754 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.374755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.375297 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.375298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.375646 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.375646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.375687 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.376165 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:11.376170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:11.378053 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
W20260620 00:13:11.378268 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:11.373526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33583 (local address 127.0.98.62:45767)
0620 00:13:11.373674 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:11.373676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:11.373685 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:11.373711 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:11.373712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:11.373722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:11.373788 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:11.374149 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.374662 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.375370 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.375558 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.375633 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.378179 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33583: BlockingRecv error: recv got EOF from 127.0.98.1:33583 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:11.521317 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:11.516122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:36285)
0620 00:13:11.516296 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:11.516320 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:11.516389 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:11.516531 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:11.516532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:11.516736 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:11.516798 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.516801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.517666 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.517667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.518213 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.518215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.518632 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.518633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.518684 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.519240 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:11.519245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:11.521244 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
W20260620 00:13:11.521452 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:11.516289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36285 (local address 127.0.98.62:45767)
0620 00:13:11.516370 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:11.516372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:11.516382 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:13:11.516447 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:11.516448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:11.516458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:11.516508 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:11.516892 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.517581 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.518336 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.518529 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.518599 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.521364 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36285: BlockingRecv error: recv got EOF from 127.0.98.2:36285 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:11 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:11.650106 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:11.645304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:53387)
0620 00:13:11.645452 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:11.645491 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:11.645571 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:11.645750 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:11.645751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:11.645924 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:11.645969 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.645971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.646639 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.646640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.647158 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:11.647160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.647528 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.647529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:11.647587 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.648056 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:11.648060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:11.650006 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
W20260620 00:13:11.650158 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:11.645436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53387 (local address 127.0.98.62:45767)
0620 00:13:11.645518 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:11.645520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:11.645550 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:13:11.645677 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:11.645678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:11.645693 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:11.645745 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:11.646043 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.646546 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.647241 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:11.647438 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:11.647525 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:11.650084 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53387: BlockingRecv error: recv got EOF from 127.0.98.3:53387 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:12.389376 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:12.378690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:54599)
0620 00:13:12.378830 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:12.378843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:12.378909 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:12.379129 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:12.379130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:12.379296 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:12.379368 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.379369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.380213 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.380214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.381013 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.381016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.381520 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.381521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.381567 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.387243 (+ 5676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:12.387249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:12.389312 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260620 00:13:12.389567 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:12.378811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54599 (local address 127.0.98.62:45767)
0620 00:13:12.378948 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:12.378950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:12.378962 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:12.379000 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:12.379000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:12.379012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:12.379093 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:12.379467 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.380122 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.381147 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.381440 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.381506 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.389449 (+ 7943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54599: BlockingRecv error: recv got EOF from 127.0.98.1:54599 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:12.527303 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:12.521931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:34977)
0620 00:13:12.522091 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:12.522104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:12.522166 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:12.522326 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:12.522326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:12.522537 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:12.522601 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.522604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.523595 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.523596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.524123 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.524124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.524680 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.524681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.524747 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.525301 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:12.525309 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:12.527242 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
W20260620 00:13:12.527503 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:12.522070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34977 (local address 127.0.98.62:45767)
0620 00:13:12.522141 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:12.522143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:12.522156 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:13:12.522218 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:12.522219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:12.522230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:12.522296 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:12.522711 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.523515 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.524267 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.524455 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.524522 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.527423 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34977: BlockingRecv error: recv got EOF from 127.0.98.2:34977 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:12 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:12.655493 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:12.650681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:54397)
0620 00:13:12.650847 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:12.650858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:12.650923 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:12.651053 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:12.651053 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:12.651212 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:12.651262 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.651264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.651894 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.651895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.652449 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:12.652451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.652793 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.652793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:12.652835 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.653365 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:12.653370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:12.655394 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
W20260620 00:13:12.655537 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:12.650754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54397 (local address 127.0.98.62:45767)
0620 00:13:12.650807 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:12.650809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:12.650910 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:13:12.650980 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:12.650981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:12.650992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:12.651033 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:12.651331 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.651824 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.652527 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:12.652727 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:12.652782 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:12.655473 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54397: BlockingRecv error: recv got EOF from 127.0.98.3:54397 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:13.395483 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:13.390009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:47023)
0620 00:13:13.390248 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:13.390260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:13.390330 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:13.390471 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:13.390472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:13.390637 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:13.390714 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.390715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.391425 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.391426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.392049 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.392051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.392719 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.392720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.392767 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.393380 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:13.393385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:13.395380 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260620 00:13:13.395541 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:13.390119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47023 (local address 127.0.98.62:45767)
0620 00:13:13.390248 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:13.390258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:13.390328 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:13:13.390389 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:13.390390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:13.390401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:13.390454 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:13.390827 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.391339 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.392151 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.392619 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.392694 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.395464 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47023: BlockingRecv error: recv got EOF from 127.0.98.1:47023 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":118,"threads_started":1}
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:13.532948 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:13.527935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:53817)
0620 00:13:13.528086 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:13.528100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:13.528165 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:13.528429 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:13.528431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:13.528713 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:13.528777 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.528779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.529460 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.529461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.529992 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.529994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.530349 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.530350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.530394 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.530857 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:13.530862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:13.532856 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
W20260620 00:13:13.533106 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:13.528124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53817 (local address 127.0.98.62:45767)
0620 00:13:13.528204 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:13.528206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:13.528217 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:13:13.528298 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:13.528299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:13.528313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:13.528373 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:13.528867 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.529382 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.530062 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.530281 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.530335 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.533010 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53817: BlockingRecv error: recv got EOF from 127.0.98.2:53817 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:13 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:13.660784 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:13.656013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:53551)
0620 00:13:13.656284 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:13.656295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:13.656362 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:13.656579 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:13.656581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:13.656758 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:13.656806 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.656808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.657449 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.657449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.657973 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:13.657974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.658280 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.658281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:13.658318 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.658761 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:13.658765 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:13.660684 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":173,"threads_started":1}
W20260620 00:13:13.660880 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:13.656078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53551 (local address 127.0.98.62:45767)
0620 00:13:13.656120 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:13.656122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:13.656363 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:13:13.656406 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:13.656406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:13.656419 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:13.656509 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:13.656877 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.657366 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.658043 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:13.658217 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:13.658269 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:13.660763 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53551: BlockingRecv error: recv got EOF from 127.0.98.3:53551 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:14.400820 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:14.395935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:33029)
0620 00:13:14.396108 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:14.396119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:14.396182 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:14.396319 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:14.396320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:14.396481 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:14.396529 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.396531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.397194 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.397195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.397745 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.397747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.398138 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.398139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.398191 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.398733 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:14.398738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:14.400726 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
W20260620 00:13:14.400914 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:14.396022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33029 (local address 127.0.98.62:45767)
0620 00:13:14.396123 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:14.396124 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:14.396180 (+ 56us) server_negotiation.cc:407] Connection header received
0620 00:13:14.396223 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:14.396224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:14.396234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:14.396293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:14.396606 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.397116 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.397841 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.398049 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.398103 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.400839 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33029: BlockingRecv error: recv got EOF from 127.0.98.1:33029 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:14.538525 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:14.533561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:49987)
0620 00:13:14.533700 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:14.533712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:14.533780 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:14.533987 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:14.533988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:14.534152 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:14.534202 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.534203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.534898 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.534899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.535475 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.535477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.535876 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.535877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.535927 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.536372 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:14.536376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:14.538463 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260620 00:13:14.538695 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:14.533622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49987 (local address 127.0.98.62:45767)
0620 00:13:14.533689 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:14.533691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:14.533783 (+ 92us) server_negotiation.cc:407] Connection header received
0620 00:13:14.533856 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:14.533856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:14.533867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:14.533935 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:14.534320 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.534818 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.535579 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.535785 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.535851 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.538590 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49987: BlockingRecv error: recv got EOF from 127.0.98.2:49987 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:14 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:14.666275 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:14.661297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:59881)
0620 00:13:14.661477 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:14.661488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:14.661559 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:14.661730 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:14.661731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:14.661915 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:14.661963 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.661964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.662669 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.662671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.663207 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:14.663209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.663616 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.663617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:14.663673 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.664145 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:14.664150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:14.666151 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
W20260620 00:13:14.666302 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:14.661381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59881 (local address 127.0.98.62:45767)
0620 00:13:14.661457 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:14.661460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:14.661546 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:13:14.661631 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:14.661631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:14.661643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:14.661721 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:14.662050 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.662575 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.663327 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:14.663530 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:14.663614 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:14.666235 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59881: BlockingRecv error: recv got EOF from 127.0.98.3:59881 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:15.406965 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:15.401420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:59339)
0620 00:13:15.401601 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:15.401614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:15.401676 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:15.401968 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:15.401969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:15.402264 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:15.402341 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.402344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.403140 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.403141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.403715 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.403717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.404095 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.404096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.404136 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.404681 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:15.404687 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:15.406863 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":124,"threads_started":1}
W20260620 00:13:15.407014 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:15.401570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59339 (local address 127.0.98.62:45767)
0620 00:13:15.401687 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:15.401689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:15.401698 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:15.401799 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:15.401800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:15.401843 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:15.401915 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:15.402493 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.403067 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.403807 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.404015 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.404081 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.406943 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59339: BlockingRecv error: recv got EOF from 127.0.98.1:59339 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:15.543864 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:15.539133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:55127)
0620 00:13:15.539293 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:15.539304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:15.539374 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:15.539509 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:15.539510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:15.539698 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:15.539747 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.539748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.540486 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.540486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.541042 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.541044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.541405 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.541406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.541444 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.541965 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:15.541970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:15.543792 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
W20260620 00:13:15.543996 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:15.539190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55127 (local address 127.0.98.62:45767)
0620 00:13:15.539243 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:15.539246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:15.539409 (+ 163us) server_negotiation.cc:407] Connection header received
0620 00:13:15.539435 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:15.539436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:15.539447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:15.539488 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:15.539887 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.540406 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.541138 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.541327 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.541400 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.543906 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55127: BlockingRecv error: recv got EOF from 127.0.98.2:55127 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:15 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:15.671927 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:15.666911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:54077)
0620 00:13:15.667063 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:15.667078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:15.667178 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:15.667423 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:15.667424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:15.667632 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:15.667677 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.667679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.668365 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.668367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.668973 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:15.668975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.669350 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.669351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:15.669407 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.669897 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:15.669902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:15.671825 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260620 00:13:15.671983 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:15.666946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54077 (local address 127.0.98.62:45767)
0620 00:13:15.667068 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:15.667071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:15.667159 (+ 88us) server_negotiation.cc:407] Connection header received
0620 00:13:15.667266 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:15.667267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:15.667280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:15.667409 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:15.667759 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.668257 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.669066 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:15.669250 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:15.669351 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:15.671916 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54077: BlockingRecv error: recv got EOF from 127.0.98.3:54077 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:16.412424 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:16.407473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:38221)
0620 00:13:16.407611 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:16.407622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:16.407682 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:16.407899 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:16.407900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:16.408110 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:16.408159 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.408161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.408890 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.408891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.409526 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.409528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.409903 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.409904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.409953 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.410416 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:16.410420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:16.412313 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
W20260620 00:13:16.412478 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:16.407630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38221 (local address 127.0.98.62:45767)
0620 00:13:16.407781 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:16.407782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:16.407790 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:13:16.407810 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:16.407811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:16.407821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:16.407884 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:16.408262 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.408814 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.409608 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.409813 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.409875 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.412403 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38221: BlockingRecv error: recv got EOF from 127.0.98.1:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:16.550166 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:16.544486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:37903)
0620 00:13:16.544722 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:16.544738 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:16.544846 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:16.545017 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:16.545018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:16.545233 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:16.545278 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.545279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.546110 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.546111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.546649 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.546650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.547060 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.547061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.547136 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.547715 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:16.547720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:16.550106 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":151,"threads_started":1}
W20260620 00:13:16.550319 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:16.544660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37903 (local address 127.0.98.62:45767)
0620 00:13:16.544738 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:16.544741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:16.544825 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:13:16.544917 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:16.544918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:16.544932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:16.545011 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:16.545364 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.546021 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.546755 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.546967 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.547039 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.550222 (+ 3183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37903: BlockingRecv error: recv got EOF from 127.0.98.2:37903 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:16 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:16.677328 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:16.672501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:55551)
0620 00:13:16.672671 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:16.672684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:16.672758 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:16.672953 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:16.672954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:16.673145 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:16.673191 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.673193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.673864 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.673866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.674392 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:16.674394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.674791 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.674792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:16.674855 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.675332 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:16.675336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:16.677229 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
W20260620 00:13:16.677379 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:16.672617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55551 (local address 127.0.98.62:45767)
0620 00:13:16.672681 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:16.672683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:16.672757 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:13:16.672860 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:16.672861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:16.672872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:16.672944 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:16.673280 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.673778 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.674484 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:16.674680 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:16.674776 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:16.677312 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55551: BlockingRecv error: recv got EOF from 127.0.98.3:55551 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:17.422046 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:17.412994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:33437)
0620 00:13:17.413138 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:17.413149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:17.413200 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:17.413433 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:17.413439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:17.413626 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:17.413684 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.413687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.414394 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.414395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.415190 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.415193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.415585 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.415586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.415635 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.419662 (+ 4027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:17.419674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:17.421983 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260620 00:13:17.422225 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:17.413066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33437 (local address 127.0.98.62:45767)
0620 00:13:17.413240 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:17.413241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:17.413251 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:13:17.413275 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:17.413275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:17.413284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:17.413356 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:17.413794 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.414308 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.415295 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.415493 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.415561 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.422123 (+ 6562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33437: BlockingRecv error: recv got EOF from 127.0.98.1:33437 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:17.555742 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:17.550831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:59481)
0620 00:13:17.551025 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:17.551038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:17.551104 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:17.551231 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:17.551232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:17.551391 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:17.551440 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.551442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.552109 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.552110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.552676 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.552678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.553002 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.553003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.553061 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.553731 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:17.553736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:17.555680 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":96,"threads_started":1}
W20260620 00:13:17.555902 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:17.550979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59481 (local address 127.0.98.62:45767)
0620 00:13:17.551048 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:17.551050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:17.551089 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:13:17.551153 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:17.551154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:17.551164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:17.551227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:17.551526 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.552030 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.552764 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.552945 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.553033 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.555827 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59481: BlockingRecv error: recv got EOF from 127.0.98.2:59481 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:17 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:17.682778 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:17.677962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:47199)
0620 00:13:17.678108 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:17.678124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:17.678177 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:17.678313 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:17.678313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:17.678479 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:17.678530 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.678531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.679194 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.679194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.679732 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:17.679734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.680129 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.680130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:17.680176 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.680776 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:17.680781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:17.682716 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260620 00:13:17.682942 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:17.678071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47199 (local address 127.0.98.62:45767)
0620 00:13:17.678168 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:17.678170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:17.678185 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:13:17.678219 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:17.678219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:17.678230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:17.678292 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:17.678593 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.679094 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.679828 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:17.680036 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:17.680105 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:17.682843 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47199: BlockingRecv error: recv got EOF from 127.0.98.3:47199 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:18.427816 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:18.422738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:33833)
0620 00:13:18.422939 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:18.422955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:18.423032 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:18.423183 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:18.423184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:18.423354 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:18.423403 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.423404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.424178 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.424179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.424740 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.424742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.425089 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.425090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.425145 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.425627 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:18.425631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:18.427744 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
W20260620 00:13:18.427963 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:18.422879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33833 (local address 127.0.98.62:45767)
0620 00:13:18.423018 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:18.423019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:18.423030 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:13:18.423094 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:18.423095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:18.423105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:18.423156 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:18.423511 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.424087 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.424831 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.425024 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.425078 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.427861 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33833: BlockingRecv error: recv got EOF from 127.0.98.1:33833 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:18.561604 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:18.556280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:39359)
0620 00:13:18.556463 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:18.556475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:18.556589 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:18.556760 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:18.556761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:18.556926 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:18.556977 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.556978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.557759 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.557761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.558321 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.558324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.558767 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.558768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.558821 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.559302 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:18.559307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:18.561528 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
W20260620 00:13:18.561748 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:18.556349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39359 (local address 127.0.98.62:45767)
0620 00:13:18.556422 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:18.556424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:18.556576 (+ 152us) server_negotiation.cc:407] Connection header received
0620 00:13:18.556685 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:18.556686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:18.556698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:18.556753 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:18.557079 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.557614 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.558424 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.558674 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.558737 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.561621 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39359: BlockingRecv error: recv got EOF from 127.0.98.2:39359 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:18 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:18.688194 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:18.683322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:39159)
0620 00:13:18.683484 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:18.683515 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:18.683581 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:18.683766 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:18.683766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:18.683947 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:18.683992 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.683994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.684759 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.684760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.685287 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:18.685289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.685676 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.685677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:18.685742 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.686228 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:18.686232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:18.688070 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
W20260620 00:13:18.688223 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:18.683478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39159 (local address 127.0.98.62:45767)
0620 00:13:18.683541 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:18.683543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:18.683587 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:13:18.683667 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:18.683668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:18.683688 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:18.683755 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:18.684073 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.684676 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.685377 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:18.685580 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:18.685674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:18.688162 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39159: BlockingRecv error: recv got EOF from 127.0.98.3:39159 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:19.433764 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:19.428480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:34279)
0620 00:13:19.428686 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:19.428700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:19.428798 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:19.428941 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:19.428941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:19.429108 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:19.429157 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.429158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.429944 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.429945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.430499 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.430502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.430947 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.430948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.430993 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.431493 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:19.431498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:19.433679 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
W20260620 00:13:19.433923 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:19.428639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34279 (local address 127.0.98.62:45767)
0620 00:13:19.428807 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:19.428809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:19.428822 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:13:19.428848 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:19.428849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:19.428860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:19.428921 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:19.429266 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.429839 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.430603 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.430852 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.430919 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.433837 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34279: BlockingRecv error: recv got EOF from 127.0.98.1:34279 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:19.566911 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:19.562095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:44737)
0620 00:13:19.562297 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:19.562312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:19.562416 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:19.562592 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:19.562594 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:19.562828 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:19.562877 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.562878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.563569 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.563570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.564113 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.564115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.564484 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.564485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.564574 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.565071 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:19.565082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:19.566813 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":89,"threads_started":1}
W20260620 00:13:19.566954 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:19.562193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44737 (local address 127.0.98.62:45767)
0620 00:13:19.562302 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:19.562304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:19.562396 (+ 92us) server_negotiation.cc:407] Connection header received
0620 00:13:19.562486 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:19.562487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:19.562511 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:19.562586 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:19.562962 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.563472 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.564202 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.564407 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.564486 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.566892 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44737: BlockingRecv error: recv got EOF from 127.0.98.2:44737 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:19 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:19.694617 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:19.688922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:40439)
0620 00:13:19.689243 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:19.689258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:19.689369 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:19.689560 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:19.689561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:19.689827 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:19.689889 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.689891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.690748 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.690749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.691356 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:19.691358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.691777 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.691778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:19.691847 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.692429 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:19.692435 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:19.694507 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":106,"threads_started":1}
W20260620 00:13:19.694666 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:19.689061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40439 (local address 127.0.98.62:45767)
0620 00:13:19.689245 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:19.689248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:19.689348 (+ 100us) server_negotiation.cc:407] Connection header received
0620 00:13:19.689446 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:19.689447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:19.689473 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:19.689553 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:19.689991 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.690592 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.691459 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:19.691675 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:19.691774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:19.694596 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40439: BlockingRecv error: recv got EOF from 127.0.98.3:40439 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:20.439684 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:20.434471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:54929)
0620 00:13:20.434682 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:20.434695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:20.434816 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:20.434993 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:20.434994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:20.435171 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:20.435235 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.435237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.435975 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.435976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.436581 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.436584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.436986 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.436987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.437039 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.437839 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:20.437844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:20.439624 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
W20260620 00:13:20.439855 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:20.434607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54929 (local address 127.0.98.62:45767)
0620 00:13:20.434734 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:20.434736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:20.434820 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:13:20.434865 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:20.434865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:20.434877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:20.434939 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:20.435352 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.435872 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.436645 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.436873 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.436939 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.439748 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54929: BlockingRecv error: recv got EOF from 127.0.98.1:54929 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:20.572504 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:20.567481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:50453)
0620 00:13:20.567657 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:20.567678 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:20.567756 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:20.567986 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:20.567987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:20.568155 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:20.568215 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.568217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.568967 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.568968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.569529 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.569531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.569864 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.569864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.569912 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.570432 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:20.570438 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:20.572443 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
W20260620 00:13:20.572748 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:20.567618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50453 (local address 127.0.98.62:45767)
0620 00:13:20.567690 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:20.567692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:20.567853 (+ 161us) server_negotiation.cc:407] Connection header received
0620 00:13:20.567886 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:20.567886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:20.567897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:20.567956 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:20.568329 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.568849 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.569606 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.569793 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.569843 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.572680 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50453: BlockingRecv error: recv got EOF from 127.0.98.2:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:20 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:20.700089 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:20.695172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:49577)
0620 00:13:20.695311 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:20.695324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:20.695398 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:20.695579 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:20.695580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:20.695772 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:20.695817 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.695819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.696486 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.696487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.697092 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:20.697094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.697536 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.697537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:20.697618 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.698172 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:20.698177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:20.700030 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260620 00:13:20.700194 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:20.695273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49577 (local address 127.0.98.62:45767)
0620 00:13:20.695348 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:20.695350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:20.695379 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:13:20.695479 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:20.695480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:20.695494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:20.695572 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:20.695905 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.696405 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.697211 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:20.697419 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:20.697504 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:20.700125 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49577: BlockingRecv error: recv got EOF from 127.0.98.3:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:21.445641 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:21.440378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:37731)
0620 00:13:21.440685 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:21.440697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:21.440774 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:21.440917 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:21.440917 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:21.441119 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:21.441170 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.441172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.442092 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.442094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.442638 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.442640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.443069 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.443070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.443113 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.443662 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:21.443668 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:21.445578 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":226,"threads_started":1}
W20260620 00:13:21.445823 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:21.440600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37731 (local address 127.0.98.62:45767)
0620 00:13:21.440708 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:21.440711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:21.440772 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:13:21.440822 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:21.440823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:21.440833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:21.440895 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:21.441281 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.442016 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.442731 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.442977 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.443037 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.445735 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37731: BlockingRecv error: recv got EOF from 127.0.98.1:37731 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:21.577970 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:21.573134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:52517)
0620 00:13:21.573300 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:21.573315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:21.573385 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:21.573535 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:21.573536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:21.573695 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:21.573743 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.573745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.574414 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.574415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.574939 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.574940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.575300 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.575302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.575348 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.575919 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:21.575925 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:21.577900 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
W20260620 00:13:21.578282 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:21.573283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52517 (local address 127.0.98.62:45767)
0620 00:13:21.573353 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:21.573355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:21.573368 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:13:21.573449 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:21.573450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:21.573461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:21.573511 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:21.573807 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.574331 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.575010 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.575227 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.575282 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.578204 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52517: BlockingRecv error: recv got EOF from 127.0.98.2:52517 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:21 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:21.705521 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:21.700632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:54603)
0620 00:13:21.700778 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:21.700788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:21.700889 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:21.701042 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:21.701043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:21.701203 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:21.701251 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.701252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.702023 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.702024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.702599 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:21.702602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.702977 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.702978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:21.703029 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.703496 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:21.703502 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:21.705450 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260620 00:13:21.705672 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:21.700718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54603 (local address 127.0.98.62:45767)
0620 00:13:21.700788 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:21.700793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:21.700889 (+ 96us) server_negotiation.cc:407] Connection header received
0620 00:13:21.700956 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:21.700957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:21.700971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:21.701017 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:21.701371 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.701903 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.702691 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:21.702878 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:21.702939 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:21.705581 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54603: BlockingRecv error: recv got EOF from 127.0.98.3:54603 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:22.456158 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:22.446221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:34855)
0620 00:13:22.446404 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:22.446415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:22.446486 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:22.446670 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:22.446672 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:22.446839 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:22.446887 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.446889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.447666 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.447668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.448428 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.448430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.448822 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.448823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.448881 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.453457 (+ 4576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:22.453464 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:22.456098 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":92,"threads_started":1}
W20260620 00:13:22.456357 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:22.446368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34855 (local address 127.0.98.62:45767)
0620 00:13:22.446496 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:22.446498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:22.446510 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:22.446537 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:22.446537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:22.446547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:22.446615 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:22.446977 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.447556 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.448509 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.448744 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.448824 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.456229 (+ 7405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34855: BlockingRecv error: recv got EOF from 127.0.98.1:34855 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:22.583288 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:22.578473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:54105)
0620 00:13:22.578625 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:22.578646 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:22.578707 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:22.578849 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:22.578850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:22.579006 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:22.579055 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.579057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.580006 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.580007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.580535 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.580571 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.580926 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.580927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.580965 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.581403 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:22.581407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:22.583218 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260620 00:13:22.583356 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:22.578586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54105 (local address 127.0.98.62:45767)
0620 00:13:22.578649 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:22.578651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:22.578705 (+ 54us) server_negotiation.cc:407] Connection header received
0620 00:13:22.578755 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:22.578755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:22.578766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:22.578826 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:22.579149 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.579934 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.580664 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.580851 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.580928 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.583310 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54105: BlockingRecv error: recv got EOF from 127.0.98.2:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:22 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:22.710881 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:22.706082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.3:42881)
0620 00:13:22.706228 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:22.706247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:22.706305 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:22.706441 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:22.706442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:22.706601 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:22.706650 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.706651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.707300 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.707301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.707846 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:22.707849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.708239 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.708240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:22.708301 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.708816 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:22.708821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:22.710822 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
W20260620 00:13:22.711078 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:22.706171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42881 (local address 127.0.98.62:45767)
0620 00:13:22.706230 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:22.706232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:22.706303 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:13:22.706349 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:22.706349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:22.706362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:22.706421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:22.706710 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.707223 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.707959 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:22.708152 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:22.708238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:22.711001 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42881: BlockingRecv error: recv got EOF from 127.0.98.3:42881 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 00:13:23 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:23 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:23.461982 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:23.456733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.1:48225)
0620 00:13:23.456850 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:23.456869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:23.456917 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:23.457071 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:23.457072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:23.457232 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:23.457282 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.457284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.458038 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.458039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.458627 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.458629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.459126 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.459128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.459176 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.459649 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:23.459654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:23.461918 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
W20260620 00:13:23.462217 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:23.456778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48225 (local address 127.0.98.62:45767)
0620 00:13:23.456899 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:23.456901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:23.456955 (+ 54us) server_negotiation.cc:407] Connection header received
0620 00:13:23.456979 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:23.456980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:23.456994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:23.457065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:23.457384 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.457934 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.458716 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.459004 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.459071 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.462135 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48225: BlockingRecv error: recv got EOF from 127.0.98.1:48225 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
Jun 20 00:13:23 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:13:23 dist-test-slave-28bv krb5kdc[8567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:13:23.588855 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:23.583947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45767 (local address 127.0.98.2:41695)
0620 00:13:23.584128 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:23.584141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:23.584201 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:23.584342 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:23.584343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:23.584503 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:23.584618 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.584620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.585343 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.585344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.585891 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.585893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.586227 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.586228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.586265 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.586750 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:23.586754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:23.588795 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45767: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
W20260620 00:13:23.588925 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:23.584117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41695 (local address 127.0.98.62:45767)
0620 00:13:23.584183 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:23.584185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:23.584196 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:13:23.584266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:23.584267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:23.584276 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:23.584317 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:23.584750 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.585262 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.585965 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.586161 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.586218 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.588879 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41695: BlockingRecv error: recv got EOF from 127.0.98.2:41695 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:13:23.620690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9150
I20260620 00:13:23.625401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9303
I20260620 00:13:23.629940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9436
I20260620 00:13:23.634526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9080
2026-06-20T00:13:23Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17537 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T00:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:23Z Disabled control of system clock
I20260620 00:13:23.662207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35759
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:13:23.743325 9632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:23.743538 9632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:23.743558 9632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:23.744755 9632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:23.744800 9632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:23.744813 9632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:23.744825 9632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:23.744837 9632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:23.746168 9632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:23.746376 9632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:23.746622 9632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:23.751684 9638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:23.751701 9640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:23.751761 9632 server_base.cc:1034] running on GCE node
W20260620 00:13:23.751691 9637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:23.752326 9632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:23.752681 9632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:23.753867 9632 hybrid_clock.cc:648] HybridClock initialized: now 1781914403753860 us; error 38 us; skew 500 ppm
I20260620 00:13:23.755237 9632 webserver.cc:460] Webserver started at http://127.0.98.62:46773/ using document root <none> and password file <none>
I20260620 00:13:23.755503 9632 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:23.755553 9632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:23.755712 9632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:23.756994 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "1a457ba371524139a60f2b644046c1e6"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "01972d5914c343392396f823dcde041e"
server_key_iv: "323db2223b9d55901c4c7e0efddd64cb"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.757357 9632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a457ba371524139a60f2b644046c1e6"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "01972d5914c343392396f823dcde041e"
server_key_iv: "323db2223b9d55901c4c7e0efddd64cb"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.758736 9632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:13:23.759521 9646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:23.759764 9632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:23.759856 9632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "1a457ba371524139a60f2b644046c1e6"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "01972d5914c343392396f823dcde041e"
server_key_iv: "323db2223b9d55901c4c7e0efddd64cb"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.759934 9632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:23.783036 9632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:23.783365 9632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:23.783533 9632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:23.787472 9632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35759
I20260620 00:13:23.787521 9698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35759 every 8 connection(s)
I20260620 00:13:23.787899 9632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:23.788576 9699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:23.791122 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6: Bootstrap starting.
I20260620 00:13:23.791739 9699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:23.792054 9699 log.cc:824] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:23.792748 9699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6: No bootstrap required, opened a new log
I20260620 00:13:23.794279 9699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } }
I20260620 00:13:23.794461 9699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:23.794487 9699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a457ba371524139a60f2b644046c1e6, State: Initialized, Role: FOLLOWER
I20260620 00:13:23.794673 9699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } }
I20260620 00:13:23.794739 9699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:23.794782 9699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:23.794831 9699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:23.795370 9699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } }
I20260620 00:13:23.795481 9699 leader_election.cc:302] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a457ba371524139a60f2b644046c1e6; no voters:
I20260620 00:13:23.795673 9699 leader_election.cc:288] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:13:23.795702 9702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:23.795866 9702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [term 1 LEADER]: Becoming Leader. State: Replica: 1a457ba371524139a60f2b644046c1e6, State: Running, Role: LEADER
I20260620 00:13:23.795938 9699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:23.796061 9702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } }
I20260620 00:13:23.796507 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a457ba371524139a60f2b644046c1e6. Latest consensus state: current_term: 1 leader_uuid: "1a457ba371524139a60f2b644046c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } } }
I20260620 00:13:23.796628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9632
I20260620 00:13:23.796658 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a457ba371524139a60f2b644046c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a457ba371524139a60f2b644046c1e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35759 } } }
I20260620 00:13:23.796689 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:23.796713 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:23.796721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:13:23.796932 392 external_mini_cluster.cc:1428] Setting key 2bbd07733ee9691309bcd209f6f42e34
I20260620 00:13:23.797102 9707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:23.797734 9707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:23.800329 9707 catalog_manager.cc:1347] Generated new cluster ID: 16195ea23a1e44d6959bcaa75775207f
I20260620 00:13:23.800386 9707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:13:23.803108 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:23.797614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.0.98.62:35759)
0620 00:13:23.797805 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:23.797814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:23.797843 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:13:23.797956 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:23.797963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:23.798029 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:23.798398 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:23.798511 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.799850 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.800945 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.801591 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.801724 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.802415 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:23.802419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:23.802420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:23.802632 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:23.802672 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:23.802687 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:23.802760 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:23.802825 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 00:13:23.802901 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260620 00:13:23.828109 9707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:13:23.828739 9707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:23.837195 9707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a457ba371524139a60f2b644046c1e6: Generated new TSK 0
I20260620 00:13:23.837432 9707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:13:23.841997 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 00:13:23.929397 9723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:23.929615 9723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:23.929636 9723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:23.930738 9723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:23.930771 9723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:23.930795 9723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:23.932036 9723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:23.932229 9723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:23.932475 9723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:23.937448 9731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:23.937505 9728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:23.937575 9723 server_base.cc:1034] running on GCE node
W20260620 00:13:23.937464 9729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:23.937968 9723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:23.938287 9723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:23.939458 9723 hybrid_clock.cc:648] HybridClock initialized: now 1781914403939460 us; error 43 us; skew 500 ppm
I20260620 00:13:23.940919 9723 webserver.cc:460] Webserver started at http://127.0.98.1:35773/ using document root <none> and password file <none>
I20260620 00:13:23.941192 9723 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:23.941267 9723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:23.941427 9723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:23.942632 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "b30ab0d4c8e24a77a3e54094d2e79781"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "dfcda696bd998c74fff78e66c57bf1c9"
server_key_iv: "f7bf611ac691dbcf5464b806bc220a70"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.943030 9723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b30ab0d4c8e24a77a3e54094d2e79781"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "dfcda696bd998c74fff78e66c57bf1c9"
server_key_iv: "f7bf611ac691dbcf5464b806bc220a70"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.944422 9723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:13:23.945467 9737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:23.945673 9723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:23.945791 9723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "b30ab0d4c8e24a77a3e54094d2e79781"
format_stamp: "Formatted at 2026-06-20 00:13:23 on dist-test-slave-28bv"
server_key: "dfcda696bd998c74fff78e66c57bf1c9"
server_key_iv: "f7bf611ac691dbcf5464b806bc220a70"
server_key_version: "encryptionkey@0"
I20260620 00:13:23.945871 9723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:23.956426 9723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:23.956831 9723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:23.957003 9723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:23.957259 9723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:23.957623 9723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:23.957681 9723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:23.957726 9723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:23.957757 9723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:23.964337 9723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39555
I20260620 00:13:23.964375 9850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39555 every 8 connection(s)
I20260620 00:13:23.964826 9723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:13:23.967473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9723
I20260620 00:13:23.967582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:13:23.967796 392 external_mini_cluster.cc:1428] Setting key f5e78cbc97b3a65ed5dda44cef51dbe3
I20260620 00:13:23.968919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 00:13:23.970577 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:23.965605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55597 (local address 127.0.98.62:35759)
0620 00:13:23.965670 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:23.965672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:23.965886 (+ 214us) server_negotiation.cc:407] Connection header received
0620 00:13:23.966433 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:23.966434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:23.966451 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:23.966499 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:23.967216 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.967803 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.968980 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.969293 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.969369 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.970304 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:23.970306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:23.970306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:23.970311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:23.970326 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:23.970330 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:23.970384 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:23.970483 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:13:23.970514 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:13:23.970876 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:23.965578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35759 (local address 127.0.98.1:55597)
0620 00:13:23.965762 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:23.965797 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:23.966318 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:23.966588 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:23.966591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:23.966599 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:23.967097 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.967102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.967895 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.967896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.968887 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:23.968889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:23.969388 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:23.969389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:23.969523 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:23.970125 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:23.970126 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:23.970170 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:23.970205 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:23.970400 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:23.970405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:23.970409 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:23.970486 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:13:23.970605 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260620 00:13:23.971520 9851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35759
I20260620 00:13:23.971626 9851 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:23.971814 9851 heartbeater.cc:511] Master 127.0.98.62:35759 requested a full tablet report, sending...
I20260620 00:13:23.972369 9663 ts_manager.cc:194] Registered new tserver with Master: b30ab0d4c8e24a77a3e54094d2e79781 (127.0.98.1:39555)
I20260620 00:13:23.973409 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55597
W20260620 00:13:24.052078 9854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.052296 9854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.052317 9854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.053473 9854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.053519 9854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.053543 9854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:24.054870 9854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.055066 9854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.055311 9854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.060459 9859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.060490 9854 server_base.cc:1034] running on GCE node
W20260620 00:13:24.060503 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.060459 9860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.060885 9854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.061121 9854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.062299 9854 hybrid_clock.cc:648] HybridClock initialized: now 1781914404062288 us; error 29 us; skew 500 ppm
I20260620 00:13:24.063566 9854 webserver.cc:460] Webserver started at http://127.0.98.2:36501/ using document root <none> and password file <none>
I20260620 00:13:24.063810 9854 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.063879 9854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.064052 9854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.065255 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "99c359651e074632bcb9881754811d74"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "bc6732f5ef5c4b6def23c8276b1d766d"
server_key_iv: "bf25f5957d367b31f2e56dd0bc2f4dca"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.065609 9854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "99c359651e074632bcb9881754811d74"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "bc6732f5ef5c4b6def23c8276b1d766d"
server_key_iv: "bf25f5957d367b31f2e56dd0bc2f4dca"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.066887 9854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:13:24.067575 9868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.067742 9854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:13:24.067836 9854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "99c359651e074632bcb9881754811d74"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "bc6732f5ef5c4b6def23c8276b1d766d"
server_key_iv: "bf25f5957d367b31f2e56dd0bc2f4dca"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.067917 9854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.080787 9854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.081089 9854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.081238 9854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.081452 9854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:24.081782 9854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:24.081835 9854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.081877 9854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:24.081903 9854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.087846 9854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37189
I20260620 00:13:24.087893 9981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37189 every 8 connection(s)
I20260620 00:13:24.088248 9854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:13:24.093492 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.089160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40309 (local address 127.0.98.62:35759)
0620 00:13:24.089237 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.089240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.089374 (+ 134us) server_negotiation.cc:407] Connection header received
0620 00:13:24.089887 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.089888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.089897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.089950 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.090574 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.091071 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.092115 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.092316 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.092374 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:24.093255 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.093256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.093256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.093261 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.093275 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.093279 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.093324 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.093404 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.093429 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:13:24.093751 9984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.089055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35759 (local address 127.0.98.2:40309)
0620 00:13:24.089249 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:24.089286 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:24.089796 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:24.090006 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:24.090009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:24.090018 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:24.090504 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:24.090508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.091177 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.091179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:24.092025 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:24.092028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.092396 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.092397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:24.092508 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:24.093114 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:24.093115 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:24.093145 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:24.093168 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:24.093335 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:24.093339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:24.093342 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:13:24.093395 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:13:24.093478 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260620 00:13:24.094283 9982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35759
I20260620 00:13:24.094383 9982 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:24.094558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9854
I20260620 00:13:24.094604 9982 heartbeater.cc:511] Master 127.0.98.62:35759 requested a full tablet report, sending...
I20260620 00:13:24.094674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:13:24.094895 392 external_mini_cluster.cc:1428] Setting key 964d18dfc5766147c509e20d41375c47
I20260620 00:13:24.095067 9663 ts_manager.cc:194] Registered new tserver with Master: 99c359651e074632bcb9881754811d74 (127.0.98.2:37189)
I20260620 00:13:24.095778 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40309
I20260620 00:13:24.096101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 00:13:24.179800 9985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.180012 9985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.180034 9985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.181339 9985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.181384 9985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.181411 9985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:24.182757 9985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.183022 9985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.183347 9985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.188431 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.188422 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.188422 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.188748 9985 server_base.cc:1034] running on GCE node
I20260620 00:13:24.189050 9985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.189358 9985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.190560 9985 hybrid_clock.cc:648] HybridClock initialized: now 1781914404190550 us; error 51 us; skew 500 ppm
I20260620 00:13:24.191924 9985 webserver.cc:460] Webserver started at http://127.0.98.3:43261/ using document root <none> and password file <none>
I20260620 00:13:24.192189 9985 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.192265 9985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.192413 9985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.193602 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "a60386514d484d5db5bfc3b915df9c0c"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "9b3e442f541a2657603286151d42dc1d"
server_key_iv: "72f4b1a187aa88ba0ab647f63a258140"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.194002 9985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a60386514d484d5db5bfc3b915df9c0c"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "9b3e442f541a2657603286151d42dc1d"
server_key_iv: "72f4b1a187aa88ba0ab647f63a258140"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.195431 9985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:13:24.196318 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.196671 9985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:24.196776 9985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "a60386514d484d5db5bfc3b915df9c0c"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "9b3e442f541a2657603286151d42dc1d"
server_key_iv: "72f4b1a187aa88ba0ab647f63a258140"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.196866 9985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.214351 9985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.214731 9985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.214893 9985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.215144 9985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:24.215551 9985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:24.215607 9985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.215641 9985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:24.215669 9985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.221647 9985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34593
I20260620 00:13:24.221688 10112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34593 every 8 connection(s)
I20260620 00:13:24.222034 9985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:13:24.227728 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.222822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57419 (local address 127.0.98.62:35759)
0620 00:13:24.222888 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.222890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.223190 (+ 300us) server_negotiation.cc:407] Connection header received
0620 00:13:24.223697 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.223699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.223710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.223779 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.224600 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.225289 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.226335 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.226546 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.226613 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:24.227468 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.227470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.227470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.227475 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.227490 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.227494 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.227549 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.227634 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.227662 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:13:24.227934 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.222818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35759 (local address 127.0.98.3:57419)
0620 00:13:24.222996 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:24.223058 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:24.223565 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:24.223854 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:24.223857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:24.223866 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:24.224417 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:24.224422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.225395 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.225396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:24.226223 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:24.226225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:24.226631 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:24.226632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:24.226744 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:24.227303 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:24.227303 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:24.227331 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:24.227353 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:24.227558 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:24.227562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:24.227564 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:13:24.227616 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:13:24.227698 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260620 00:13:24.228497 10113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35759
I20260620 00:13:24.228611 10113 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:24.228772 10113 heartbeater.cc:511] Master 127.0.98.62:35759 requested a full tablet report, sending...
I20260620 00:13:24.229175 9663 ts_manager.cc:194] Registered new tserver with Master: a60386514d484d5db5bfc3b915df9c0c (127.0.98.3:34593)
I20260620 00:13:24.229506 9663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57419
I20260620 00:13:24.231285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 9985
I20260620 00:13:24.231395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:13:24.231609 392 external_mini_cluster.cc:1428] Setting key b1146e057e300c7d4a18ac3f3768f637
I20260620 00:13:24.232694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:13:24.234628 9715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:24.234005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.0.98.62:35759)
0620 00:13:24.234115 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.234118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.234251 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:13:24.234328 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.234329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.234342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.234386 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.234561 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38978: BlockingRecv error: recv got EOF from 127.0.0.1:38978 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:13:24.234686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9723
I20260620 00:13:24.239948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9854
I20260620 00:13:24.244920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9985
I20260620 00:13:24.249185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 9632
2026-06-20T00:13:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (613 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T00:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:24Z Disabled control of system clock
I20260620 00:13:24.276515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34301
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:13:24.357894 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.358120 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.358143 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.359297 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:24.359344 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.359359 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.359372 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:24.359385 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:24.360736 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.360949 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.361198 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.366245 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.366257 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.366264 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.366308 10124 server_base.cc:1034] running on GCE node
I20260620 00:13:24.366727 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.367038 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.368198 10124 hybrid_clock.cc:648] HybridClock initialized: now 1781914404368181 us; error 28 us; skew 500 ppm
I20260620 00:13:24.369601 10124 webserver.cc:460] Webserver started at http://127.0.98.62:40343/ using document root <none> and password file <none>
I20260620 00:13:24.369797 10124 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.369841 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.369938 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.371068 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "d3c3d371dd37416592617947e93b249e"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "7fd357c85ef894fe4e0c216031a22521"
server_key_iv: "9ab9ff239c88ee337d199d10ba02745f"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.371387 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3c3d371dd37416592617947e93b249e"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "7fd357c85ef894fe4e0c216031a22521"
server_key_iv: "9ab9ff239c88ee337d199d10ba02745f"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.372736 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:13:24.373471 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.373678 10124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:24.373770 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "d3c3d371dd37416592617947e93b249e"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "7fd357c85ef894fe4e0c216031a22521"
server_key_iv: "9ab9ff239c88ee337d199d10ba02745f"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.373826 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.377118 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.377344 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.377465 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.381198 10124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34301
I20260620 00:13:24.381243 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34301 every 8 connection(s)
I20260620 00:13:24.381547 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:24.382237 10191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:24.385017 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e: Bootstrap starting.
I20260620 00:13:24.385583 10191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:24.385876 10191 log.cc:824] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:24.386586 10191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e: No bootstrap required, opened a new log
I20260620 00:13:24.388162 10191 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } }
I20260620 00:13:24.388324 10191 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:24.388370 10191 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c3d371dd37416592617947e93b249e, State: Initialized, Role: FOLLOWER
I20260620 00:13:24.388509 10191 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } }
I20260620 00:13:24.388664 10191 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:24.388724 10191 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:24.388772 10191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:24.389284 10191 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } }
I20260620 00:13:24.389394 10191 leader_election.cc:302] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c3d371dd37416592617947e93b249e; no voters:
I20260620 00:13:24.389567 10191 leader_election.cc:288] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:13:24.389679 10194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:24.389824 10191 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:24.389858 10194 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [term 1 LEADER]: Becoming Leader. State: Replica: d3c3d371dd37416592617947e93b249e, State: Running, Role: LEADER
I20260620 00:13:24.389979 10194 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } }
I20260620 00:13:24.390543 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c3d371dd37416592617947e93b249e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } } }
I20260620 00:13:24.390630 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:24.390799 10196 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c3d371dd37416592617947e93b249e. Latest consensus state: current_term: 1 leader_uuid: "d3c3d371dd37416592617947e93b249e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c3d371dd37416592617947e93b249e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34301 } } }
I20260620 00:13:24.390870 10196 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:24.390949 10203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:24.391077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10124
I20260620 00:13:24.391161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:13:24.391382 392 external_mini_cluster.cc:1428] Setting key 55f97de274d2bed464260b4a1b880f0b
I20260620 00:13:24.391427 10203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:24.393128 10203 catalog_manager.cc:1347] Generated new cluster ID: c8c306946ce14696a507400ee4dc185e
I20260620 00:13:24.393190 10203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:13:24.393862 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.391976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.98.62:34301)
0620 00:13:24.392161 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.392169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.392200 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:13:24.392308 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.392315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.392357 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.392489 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.393418 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.393422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.393422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.393426 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.393476 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.393489 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.393549 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.393644 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.393700 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260620 00:13:24.414238 10203 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:13:24.414415 10203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:24.418243 10203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c3d371dd37416592617947e93b249e: Generated new TSK 0
I20260620 00:13:24.418434 10203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:13:24.423574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--builtin_ntp_servers=127.0.98.20:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:13:24.528439 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.528798 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.528839 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.530099 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.530176 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.530222 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:24.531662 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.532020 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.532426 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.538765 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.538727 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.538875 10215 server_base.cc:1034] running on GCE node
W20260620 00:13:24.538730 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.539260 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.539556 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.540716 10215 hybrid_clock.cc:648] HybridClock initialized: now 1781914404540699 us; error 31 us; skew 500 ppm
I20260620 00:13:24.542199 10215 webserver.cc:460] Webserver started at http://127.0.98.1:44577/ using document root <none> and password file <none>
I20260620 00:13:24.542464 10215 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.542521 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.542695 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.543936 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "70aa88265d1b48f0835f3b6c78620352"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "885fb78436da397eea19de5e7e3222c3"
server_key_iv: "15a498697e7b2707abcb5cbfa2c2a3aa"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.544328 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "70aa88265d1b48f0835f3b6c78620352"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "885fb78436da397eea19de5e7e3222c3"
server_key_iv: "15a498697e7b2707abcb5cbfa2c2a3aa"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.545753 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:13:24.546559 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.546748 10215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:24.546815 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "70aa88265d1b48f0835f3b6c78620352"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "885fb78436da397eea19de5e7e3222c3"
server_key_iv: "15a498697e7b2707abcb5cbfa2c2a3aa"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.546903 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.550367 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.550637 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.550771 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.550999 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:24.551296 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:24.551328 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.551380 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:24.551406 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.557504 10215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45337
I20260620 00:13:24.557592 10342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45337 every 8 connection(s)
I20260620 00:13:24.557927 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:13:24.559141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10215
I20260620 00:13:24.559224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:13:24.559432 392 external_mini_cluster.cc:1428] Setting key a2759dae1cf01354c033f474541808e9
I20260620 00:13:24.560612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--builtin_ntp_servers=127.0.98.20:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 00:13:24.561442 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.558682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46019 (local address 127.0.98.62:34301)
0620 00:13:24.558772 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.558775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.558960 (+ 185us) server_negotiation.cc:407] Connection header received
0620 00:13:24.559573 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.559575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.559584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.559650 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.561187 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.561188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.561188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.561193 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.561209 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.561214 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.561267 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.561355 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.561371 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:13:24.561652 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.558639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34301 (local address 127.0.98.1:46019)
0620 00:13:24.558838 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:24.558873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:24.559414 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:24.559723 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:24.559727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:24.559736 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:24.560975 (+ 1239us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:24.560975 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:24.561056 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:24.561085 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:24.561276 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:24.561281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:24.561284 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:13:24.561365 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:13:24.561426 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260620 00:13:24.562242 10343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34301
I20260620 00:13:24.562358 10343 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:24.562520 10343 heartbeater.cc:511] Master 127.0.98.62:34301 requested a full tablet report, sending...
I20260620 00:13:24.563041 10155 ts_manager.cc:194] Registered new tserver with Master: 70aa88265d1b48f0835f3b6c78620352 (127.0.98.1:45337)
W20260620 00:13:24.643244 10346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.643522 10346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.643560 10346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.644848 10346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.644932 10346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.644977 10346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:24.646502 10346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.646816 10346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.647135 10346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.652431 10351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.652431 10354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.652490 10352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.652791 10346 server_base.cc:1034] running on GCE node
I20260620 00:13:24.653035 10346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.653281 10346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.654453 10346 hybrid_clock.cc:648] HybridClock initialized: now 1781914404654437 us; error 28 us; skew 500 ppm
I20260620 00:13:24.655726 10346 webserver.cc:460] Webserver started at http://127.0.98.2:44213/ using document root <none> and password file <none>
I20260620 00:13:24.655972 10346 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.656041 10346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.656209 10346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.657406 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "e481d5f8a3be4b5a91f0ec025347f0eb"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "89c5b82dc55697f015af4b41d0dc8b42"
server_key_iv: "0bd3b95c47386bc1eeca8676d2d75d2d"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.657785 10346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e481d5f8a3be4b5a91f0ec025347f0eb"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "89c5b82dc55697f015af4b41d0dc8b42"
server_key_iv: "0bd3b95c47386bc1eeca8676d2d75d2d"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.659253 10346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:13:24.660023 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.660204 10346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:13:24.660307 10346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "e481d5f8a3be4b5a91f0ec025347f0eb"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "89c5b82dc55697f015af4b41d0dc8b42"
server_key_iv: "0bd3b95c47386bc1eeca8676d2d75d2d"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.660382 10346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.663780 10346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.664064 10346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.664211 10346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.664426 10346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:24.664767 10346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:24.664820 10346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.664860 10346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:24.664887 10346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.670666 10346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34971
I20260620 00:13:24.670704 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34971 every 8 connection(s)
I20260620 00:13:24.671073 10346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:13:24.674423 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.671851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46101 (local address 127.0.98.62:34301)
0620 00:13:24.671919 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.671921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.672213 (+ 292us) server_negotiation.cc:407] Connection header received
0620 00:13:24.672806 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.672807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.672813 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.672860 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.674113 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.674115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.674115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.674120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.674139 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.674143 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.674232 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.674332 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.674349 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:13:24.674623 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.671840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34301 (local address 127.0.98.2:46101)
0620 00:13:24.672037 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:24.672070 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:24.672698 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:24.672928 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:24.672932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:24.672942 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:24.673889 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:24.673890 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:24.673920 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:24.673950 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:24.674232 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:24.674240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:24.674244 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:24.674344 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:13:24.674407 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260620 00:13:24.675149 10474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34301
I20260620 00:13:24.675297 10474 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:24.675455 10474 heartbeater.cc:511] Master 127.0.98.62:34301 requested a full tablet report, sending...
I20260620 00:13:24.675858 10155 ts_manager.cc:194] Registered new tserver with Master: e481d5f8a3be4b5a91f0ec025347f0eb (127.0.98.2:34971)
I20260620 00:13:24.675953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10346
I20260620 00:13:24.676043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:13:24.676262 392 external_mini_cluster.cc:1428] Setting key a3ef9207ef7cbdda3f85616bfaf6a168
I20260620 00:13:24.677229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--builtin_ntp_servers=127.0.98.20:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:13:24.762208 10477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:24.762432 10477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:24.762456 10477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:24.763792 10477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:24.763839 10477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:24.763865 10477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:24.765362 10477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:24.765632 10477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:24.765957 10477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:24.771368 10485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.771392 10482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:24.771405 10483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:24.771430 10477 server_base.cc:1034] running on GCE node
I20260620 00:13:24.771834 10477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:24.772096 10477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:24.773263 10477 hybrid_clock.cc:648] HybridClock initialized: now 1781914404773250 us; error 28 us; skew 500 ppm
I20260620 00:13:24.774693 10477 webserver.cc:460] Webserver started at http://127.0.98.3:35341/ using document root <none> and password file <none>
I20260620 00:13:24.774950 10477 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:24.774997 10477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:24.775182 10477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:24.776347 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "7bdcedd5911d4830974ef0337f97fd75"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "6a20a118bd361cb7d1d9fa3ad1a84f7a"
server_key_iv: "8cd6379bccfcb89c48ab95d5d0548cc3"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.776734 10477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7bdcedd5911d4830974ef0337f97fd75"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "6a20a118bd361cb7d1d9fa3ad1a84f7a"
server_key_iv: "8cd6379bccfcb89c48ab95d5d0548cc3"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.778164 10477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:24.778947 10491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.779186 10477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:24.779258 10477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "7bdcedd5911d4830974ef0337f97fd75"
format_stamp: "Formatted at 2026-06-20 00:13:24 on dist-test-slave-28bv"
server_key: "6a20a118bd361cb7d1d9fa3ad1a84f7a"
server_key_iv: "8cd6379bccfcb89c48ab95d5d0548cc3"
server_key_version: "encryptionkey@0"
I20260620 00:13:24.779310 10477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:24.783115 10477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:24.783350 10477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:24.783456 10477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:24.783615 10477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:24.783874 10477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:24.783905 10477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.783926 10477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:24.783937 10477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:24.790104 10477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40319
I20260620 00:13:24.790134 10604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40319 every 8 connection(s)
I20260620 00:13:24.790508 10477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:13:24.792423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10477
I20260620 00:13:24.792520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:13:24.792788 392 external_mini_cluster.cc:1428] Setting key 400a8b32971c369dfbf3d010fb826550
I20260620 00:13:24.793963 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.791370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59155 (local address 127.0.98.62:34301)
0620 00:13:24.791435 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.791437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.791698 (+ 261us) server_negotiation.cc:407] Connection header received
0620 00:13:24.792150 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.792151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.792157 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.792204 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.793661 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:24.793662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:24.793663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:13:24.793667 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:24.793682 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:13:24.793686 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:24.793741 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:24.793876 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 00:13:24.793892 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:13:24.794191 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:24.791371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34301 (local address 127.0.98.3:59155)
0620 00:13:24.791569 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:24.791602 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:24.792057 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:24.792300 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:24.792303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:24.792311 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:24.793494 (+ 1183us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:13:24.793495 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:24.793546 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:13:24.793581 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:24.793756 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:24.793761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:24.793764 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:13:24.793885 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 00:13:24.793949 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260620 00:13:24.794667 10605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34301
I20260620 00:13:24.794785 10605 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:24.794932 10605 heartbeater.cc:511] Master 127.0.98.62:34301 requested a full tablet report, sending...
I20260620 00:13:24.795315 10155 ts_manager.cc:194] Registered new tserver with Master: 7bdcedd5911d4830974ef0337f97fd75 (127.0.98.3:40319)
I20260620 00:13:24.804991 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:13:24.806898 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:13:24.806434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.0.98.62:34301)
0620 00:13:24.806479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:24.806481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:24.806588 (+ 107us) server_negotiation.cc:407] Connection header received
0620 00:13:24.806643 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:24.806649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:24.806654 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:24.806695 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:24.806796 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52938: BlockingRecv error: recv got EOF from 127.0.0.1:52938 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:13:24.806957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10215
I20260620 00:13:24.811550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10346
I20260620 00:13:24.815829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10477
I20260620 00:13:24.820053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10124
2026-06-20T00:13:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (570 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:13:24 dist-test-slave-28bv krb5kdc[10615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:13:24 dist-test-slave-28bv krb5kdc[10615](info): set up 2 sockets
Jun 20 00:13:24 dist-test-slave-28bv krb5kdc[10615](info): commencing operation
krb5kdc: starting...
W20260620 00:13:25.857358 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:13:25 dist-test-slave-28bv krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:13:25.980712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:26.062702 10628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:26.062912 10628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:26.062935 10628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:26.064128 10628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:13:26.064173 10628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:26.064188 10628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:26.064200 10628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:13:26.064213 10628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:13:26.065785 10628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:26.066044 10628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:26.066293 10628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:26.071354 10633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.071354 10636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.071372 10628 server_base.cc:1034] running on GCE node
W20260620 00:13:26.071358 10634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.071791 10628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:26.072146 10628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:26.073338 10628 hybrid_clock.cc:648] HybridClock initialized: now 1781914406073327 us; error 37 us; skew 500 ppm
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:26.075978 10628 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:13:26.076385 10628 webserver.cc:460] Webserver started at http://127.0.98.62:44345/ using document root <none> and password file <none>
I20260620 00:13:26.076656 10628 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:26.076710 10628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:26.076802 10628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:26.077826 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7134eb90239e4c5c9e754a7a9c6f57e4"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4b3fe2d3031fae61de2519f5448e248"
server_key_iv: "9f126bfad54f2c621ecb636796a77ffd"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.078133 10628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7134eb90239e4c5c9e754a7a9c6f57e4"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4b3fe2d3031fae61de2519f5448e248"
server_key_iv: "9f126bfad54f2c621ecb636796a77ffd"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.079427 10628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:13:26.080214 10643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.080421 10628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:26.080508 10628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7134eb90239e4c5c9e754a7a9c6f57e4"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4b3fe2d3031fae61de2519f5448e248"
server_key_iv: "9f126bfad54f2c621ecb636796a77ffd"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.080617 10628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:26.100008 10628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:26.102394 10628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:26.102515 10628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:26.106736 10628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36889
I20260620 00:13:26.106807 10695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36889 every 8 connection(s)
I20260620 00:13:26.107122 10628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:13:26.108194 10696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:26.110584 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: Bootstrap starting.
I20260620 00:13:26.111220 10696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:26.111536 10696 log.cc:824] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:26.112398 10696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: No bootstrap required, opened a new log
I20260620 00:13:26.114130 10696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260620 00:13:26.114287 10696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:26.114311 10696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7134eb90239e4c5c9e754a7a9c6f57e4, State: Initialized, Role: FOLLOWER
I20260620 00:13:26.114476 10696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260620 00:13:26.114553 10696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:13:26.114576 10696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:13:26.114632 10696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:26.115477 10696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260620 00:13:26.115566 10696 leader_election.cc:302] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7134eb90239e4c5c9e754a7a9c6f57e4; no voters:
I20260620 00:13:26.115818 10696 leader_election.cc:288] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:13:26.115901 10699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:26.116065 10699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7134eb90239e4c5c9e754a7a9c6f57e4, State: Running, Role: LEADER
I20260620 00:13:26.116107 10696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:13:26.116153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10628
I20260620 00:13:26.116266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:13:26.116169 10699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } }
I20260620 00:13:26.116485 392 external_mini_cluster.cc:1428] Setting key de99d4071a1bd0cc37c87bb57e62c862
I20260620 00:13:26.116627 10700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260620 00:13:26.116641 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7134eb90239e4c5c9e754a7a9c6f57e4. Latest consensus state: current_term: 1 leader_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7134eb90239e4c5c9e754a7a9c6f57e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36889 } } }
I20260620 00:13:26.116771 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:26.116731 10700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:13:26.117365 10707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:13:26.117908 10707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:13:26.120817 10707 catalog_manager.cc:1347] Generated new cluster ID: 503edaa1de4c4be48b6be7370bca96eb
I20260620 00:13:26.120880 10707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:26.127996 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.117281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.98.62:36889)
0620 00:13:26.117617 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.117630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.117674 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:13:26.117832 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.117840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.117916 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.118615 (+ 699us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:26.118954 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.120325 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.121378 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.121872 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.122008 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.124362 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:26.124366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:26.124367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:26.124382 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:26.126164 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.126556 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.126559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.126559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.126679 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.127086 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.127087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.127088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.127377 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:26.127561 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.127635 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.127790 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260620 00:13:26.131980 10707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:13:26.132692 10707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:13:26.140741 10707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: Generated new TSK 0
I20260620 00:13:26.140940 10707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:13:26.176271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:26.261750 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:26.261960 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:26.261984 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:26.263324 10722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:26.263371 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:26.263401 10722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:13:26.264922 10722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:26.265168 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:26.265417 10722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:26.270413 10727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.270499 10722 server_base.cc:1034] running on GCE node
W20260620 00:13:26.270534 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.270447 10728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.270797 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:26.271080 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:26.272245 10722 hybrid_clock.cc:648] HybridClock initialized: now 1781914406272226 us; error 35 us; skew 500 ppm
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:26.274843 10722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:13:26.275348 10722 webserver.cc:460] Webserver started at http://127.0.98.1:37877/ using document root <none> and password file <none>
I20260620 00:13:26.275578 10722 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:26.275622 10722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:26.275753 10722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:26.276897 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b26f666e4264ce0b2067919dd736ac6"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "a6bbc6e16dc6685c2a9deea0a8c7dd8f"
server_key_iv: "5085b3826256798675da8208237c3b03"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.277251 10722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b26f666e4264ce0b2067919dd736ac6"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "a6bbc6e16dc6685c2a9deea0a8c7dd8f"
server_key_iv: "5085b3826256798675da8208237c3b03"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.278657 10722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:26.279495 10737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.279732 10722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:26.279826 10722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "0b26f666e4264ce0b2067919dd736ac6"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "a6bbc6e16dc6685c2a9deea0a8c7dd8f"
server_key_iv: "5085b3826256798675da8208237c3b03"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.279891 10722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:26.295944 10722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:26.298020 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:26.298136 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:26.298359 10722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:26.298696 10722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:26.298730 10722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.298753 10722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:26.298802 10722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.304558 10722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44839
I20260620 00:13:26.304667 10850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44839 every 8 connection(s)
I20260620 00:13:26.305008 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:26.311986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10722
I20260620 00:13:26.312090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:13:26.312331 392 external_mini_cluster.cc:1428] Setting key 8c91eccb47ec427600b7c48a82edf7a5
I20260620 00:13:26.314956 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.305813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32885 (local address 127.0.98.62:36889)
0620 00:13:26.305861 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.305863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.306060 (+ 197us) server_negotiation.cc:407] Connection header received
0620 00:13:26.306535 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.306536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.306547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.306608 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:26.307515 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.308045 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.309120 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.309342 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.309412 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.311541 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:26.311542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:26.311543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:26.311550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:26.313337 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.313837 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.313837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.313838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.313890 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.314182 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.314183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.314183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.314287 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:26.314363 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.314726 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.314833 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:13:26.314998 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.305751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.1:32885)
0620 00:13:26.305917 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.305959 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.306451 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.306660 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.306663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.307221 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:26.307428 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.307433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.308134 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.308134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.309003 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.309005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.309431 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.309431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.309541 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.310042 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:26.310050 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:26.311442 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:26.313415 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.313423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.313425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.313758 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.313956 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.313957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.313957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.314117 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.314377 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:26.314383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:26.314559 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.314685 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.314836 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260620 00:13:26.315551 10851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260620 00:13:26.315647 10851 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:26.315845 10851 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260620 00:13:26.316362 10660 ts_manager.cc:194] Registered new tserver with Master: 0b26f666e4264ce0b2067919dd736ac6 (127.0.98.1:44839)
I20260620 00:13:26.317355 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32885
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:13:26.340219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:26.422060 10856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:26.422290 10856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:26.422317 10856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:26.423568 10856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:26.423622 10856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:26.423650 10856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:13:26.425266 10856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:26.425521 10856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:26.425766 10856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:26.431048 10864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.431058 10861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.431035 10862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.431336 10856 server_base.cc:1034] running on GCE node
I20260620 00:13:26.431535 10856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:26.431759 10856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:26.432905 10856 hybrid_clock.cc:648] HybridClock initialized: now 1781914406432895 us; error 32 us; skew 500 ppm
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:26.435441 10856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:13:26.435922 10856 webserver.cc:460] Webserver started at http://127.0.98.2:41733/ using document root <none> and password file <none>
I20260620 00:13:26.436113 10856 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:26.436154 10856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:26.436244 10856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:26.437348 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "49b6a528473e465a9a21045ea63d96d0"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "cdbb421e2bea2cfd0b00a1a7c5aa279f"
server_key_iv: "eab5560a6b7658c8bdff4391b38fb54d"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.437668 10856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "49b6a528473e465a9a21045ea63d96d0"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "cdbb421e2bea2cfd0b00a1a7c5aa279f"
server_key_iv: "eab5560a6b7658c8bdff4391b38fb54d"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.439002 10856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:13:26.439868 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.440117 10856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:13:26.440240 10856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "49b6a528473e465a9a21045ea63d96d0"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "cdbb421e2bea2cfd0b00a1a7c5aa279f"
server_key_iv: "eab5560a6b7658c8bdff4391b38fb54d"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.440357 10856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:26.450568 10856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:26.453051 10856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:26.453234 10856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:26.453490 10856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:26.453806 10856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:26.453862 10856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.453898 10856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:26.453946 10856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.459991 10856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45021
I20260620 00:13:26.460033 10984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45021 every 8 connection(s)
I20260620 00:13:26.460373 10856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:13:26.465258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10856
I20260620 00:13:26.465375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:13:26.465595 392 external_mini_cluster.cc:1428] Setting key e791683401c006d7212a8b8def800db5
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:26.471007 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.461135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51699 (local address 127.0.98.62:36889)
0620 00:13:26.461221 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.461224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.461477 (+ 253us) server_negotiation.cc:407] Connection header received
0620 00:13:26.461978 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.461979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.461989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.462043 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:26.463344 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.463875 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.464971 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.465284 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.465420 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.467849 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:26.467851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:26.467852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:26.467860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:26.469400 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.469936 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.469937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.469937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.469986 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.470374 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.470374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.470375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.470480 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:26.470531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.470877 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.470940 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:13:26.471372 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.461112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.2:51699)
0620 00:13:26.461341 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.461382 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.461887 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.462097 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.462100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.462865 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:26.463074 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.463080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.463986 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.463988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.464876 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.464878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.465449 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.465450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.465593 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.466174 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:26.466185 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:26.467732 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:26.469489 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.469496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.469499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.469853 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.470127 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.470128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.470129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.470304 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.470546 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:26.470553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:26.470739 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.470863 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.471063 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
I20260620 00:13:26.471895 10985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260620 00:13:26.472005 10985 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:26.472265 10985 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260620 00:13:26.472672 10660 ts_manager.cc:194] Registered new tserver with Master: 49b6a528473e465a9a21045ea63d96d0 (127.0.98.2:45021)
I20260620 00:13:26.473050 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51699
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:13:26.493795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:13:26.575651 10990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:13:26.575882 10990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:13:26.575909 10990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:13:26.577077 10990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:13:26.577116 10990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:13:26.577143 10990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:13:26.578686 10990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:13:26.578896 10990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:13:26.579077 10990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:13:26.584318 10996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.584393 10995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:13:26.584318 10998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:13:26.584975 10990 server_base.cc:1034] running on GCE node
I20260620 00:13:26.585199 10990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:13:26.585459 10990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:13:26.586642 10990 hybrid_clock.cc:648] HybridClock initialized: now 1781914406586628 us; error 47 us; skew 500 ppm
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:13:26.589248 10990 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:13:26.589756 10990 webserver.cc:460] Webserver started at http://127.0.98.3:32797/ using document root <none> and password file <none>
I20260620 00:13:26.590001 10990 fs_manager.cc:359] Metadata directory not provided
I20260620 00:13:26.590071 10990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:13:26.590219 10990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:13:26.591274 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "c923ee04eddb48b8a3a821ae82101274"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4dc80b458afb556d499d4a902b9ed61"
server_key_iv: "eba758335720ac971123dcd0bc7d37c9"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.591643 10990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c923ee04eddb48b8a3a821ae82101274"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4dc80b458afb556d499d4a902b9ed61"
server_key_iv: "eba758335720ac971123dcd0bc7d37c9"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.593079 10990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:13:26.593894 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.594076 10990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:13:26.594177 10990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "c923ee04eddb48b8a3a821ae82101274"
format_stamp: "Formatted at 2026-06-20 00:13:26 on dist-test-slave-28bv"
server_key: "f4dc80b458afb556d499d4a902b9ed61"
server_key_iv: "eba758335720ac971123dcd0bc7d37c9"
server_key_version: "encryptionkey@0"
I20260620 00:13:26.594261 10990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:13:26.611836 10990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:13:26.614104 10990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:13:26.614277 10990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:13:26.614559 10990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:13:26.614882 10990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:13:26.614939 10990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.614984 10990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:13:26.615010 10990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:13:26.621546 10990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43519
I20260620 00:13:26.621590 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43519 every 8 connection(s)
I20260620 00:13:26.621922 10990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[10615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:13:26.629225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 10990
I20260620 00:13:26.629317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:13:26.629554 392 external_mini_cluster.cc:1428] Setting key def6aa9e72859f7cfeb3fe832893c74b
I20260620 00:13:26.631585 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.622658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57993 (local address 127.0.98.62:36889)
0620 00:13:26.622766 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.622769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.622943 (+ 174us) server_negotiation.cc:407] Connection header received
0620 00:13:26.623490 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.623491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.623503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.623560 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:26.624511 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.625042 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.626077 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.626286 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.626356 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.628532 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:26.628533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:26.628533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:26.628605 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:26.630346 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.630754 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.630755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.630755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.630810 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.631034 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.631035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.631035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.631136 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:26.631186 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.631463 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.631522 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 00:13:26.631758 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.622580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36889 (local address 127.0.98.3:57993)
0620 00:13:26.622803 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.622857 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.623381 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.623633 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.623637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.624220 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:13:26.624410 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.624415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.625145 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.625146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.625974 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.625976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.626373 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.626374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.626493 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.627029 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:13:26.627037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:13:26.628420 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:13:26.630404 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.630411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.630413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.630687 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.630869 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:13:26.630870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:13:26.630870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:13:26.630969 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:13:26.631194 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:13:26.631198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:13:26.631348 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.631436 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.631556 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260620 00:13:26.632228 11119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36889
I20260620 00:13:26.632349 11119 heartbeater.cc:464] Registering TS with master...
I20260620 00:13:26.632529 11119 heartbeater.cc:511] Master 127.0.98.62:36889 requested a full tablet report, sending...
I20260620 00:13:26.632957 10660 ts_manager.cc:194] Registered new tserver with Master: c923ee04eddb48b8a3a821ae82101274 (127.0.98.3:43519)
I20260620 00:13:26.633373 10660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57993
I20260620 00:13:26.641435 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:13:26.648728 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.642675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.0.98.62:36889)
0620 00:13:26.642747 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.642756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.642837 (+ 81us) server_negotiation.cc:407] Connection header received
0620 00:13:26.642878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.642878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.642887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.642934 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:13:26.643524 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.644033 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.644914 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.645101 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.645167 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.646392 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:13:26.646393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:13:26.646393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:13:26.646399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:13:26.647604 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.647936 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.647937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.647937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.647987 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:13:26.648147 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:13:26.648147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:13:26.648147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:13:26.648319 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:13:26.648397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.648497 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.648627 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:13:26.650415 10660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:13:26.651965 10660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:13:26.658602 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44839 (local address 127.0.0.1:33678)
0620 00:13:26.654452 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.654493 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.654757 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.654986 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.654989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.654992 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:26.655059 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.655061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656400 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.656401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.657483 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.657485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.658337 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.658338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.658394 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.658405 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.658461 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.658497 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":88,"thread_start_us":59,"threads_started":1}
I20260620 00:13:26.658769 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43519 (local address 127.0.0.1:55754)
0620 00:13:26.654603 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.654613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.654698 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.655086 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.655087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.655089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:26.655129 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.655130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656409 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.656409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.657675 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.657678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.658560 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.658560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.658614 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.658618 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.658670 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.658709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":56,"thread_start_us":95,"threads_started":1}
I20260620 00:13:26.658624 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.98.1:44839)
0620 00:13:26.654585 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.654590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.654602 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:26.654848 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.654850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.654870 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.654930 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:26.655126 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656280 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.657567 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.658255 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.658332 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.658388 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.658492 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.658539 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260620 00:13:26.659137 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55754 (local address 127.0.98.3:43519)
0620 00:13:26.654843 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.654848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.654857 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:13:26.654892 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.654894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.654935 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.655014 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:26.655239 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656332 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.657778 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.658471 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.658539 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.658589 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.658989 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.659042 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260620 00:13:26.659667 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45021 (local address 127.0.0.1:45642)
0620 00:13:26.655217 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.655227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.655270 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.655458 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.655459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.655461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:26.655498 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.655499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656336 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.656337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.657688 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.657690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.659445 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.659447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.659508 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.659512 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.659562 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.659600 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":9,"thread_start_us":82,"threads_started":1}
I20260620 00:13:26.659706 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.654603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.98.2:45021)
0620 00:13:26.654768 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.654772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.655322 (+ 550us) server_negotiation.cc:407] Connection header received
0620 00:13:26.655366 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.655368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.655379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.655463 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:26.655541 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.656223 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.658627 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.659361 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.659427 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.659477 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.659574 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.659615 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260620 00:13:26.659662 10785 tablet_service.cc:1467] Processing CreateTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe (DEFAULT_TABLE table=test-table [id=c9464c1800ea4065b95315cb1e133c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:26.659731 11053 tablet_service.cc:1467] Processing CreateTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe (DEFAULT_TABLE table=test-table [id=c9464c1800ea4065b95315cb1e133c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:26.659988 10785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5592617e5b841de9bc48c4ee2c37dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:26.660012 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5592617e5b841de9bc48c4ee2c37dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:26.662508 11136 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Bootstrap starting.
I20260620 00:13:26.663237 11136 tablet_bootstrap.cc:654] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:26.663205 10919 tablet_service.cc:1467] Processing CreateTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe (DEFAULT_TABLE table=test-table [id=c9464c1800ea4065b95315cb1e133c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:13:26.663558 10919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5592617e5b841de9bc48c4ee2c37dbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:13:26.663784 11136 log.cc:824] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:26.664151 11137 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: Bootstrap starting.
I20260620 00:13:26.664616 11136 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: No bootstrap required, opened a new log
I20260620 00:13:26.664690 11136 ts_tablet_manager.cc:1397] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:26.664851 11137 tablet_bootstrap.cc:654] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:26.665376 11137 log.cc:824] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:26.666150 11137 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: No bootstrap required, opened a new log
I20260620 00:13:26.666227 11137 ts_tablet_manager.cc:1397] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:26.666642 11140 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Bootstrap starting.
I20260620 00:13:26.666586 11136 raft_consensus.cc:348] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.667080 11136 raft_consensus.cc:374] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:26.667119 11136 raft_consensus.cc:729] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c923ee04eddb48b8a3a821ae82101274, State: Initialized, Role: FOLLOWER
I20260620 00:13:26.667268 11136 consensus_queue.cc:260] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.667333 11140 tablet_bootstrap.cc:654] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Neither blocks nor log segments found. Creating new log.
I20260620 00:13:26.667584 11136 ts_tablet_manager.cc:1428] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:13:26.667742 11119 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260620 00:13:26.668140 11140 log.cc:824] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:13:26.668215 11137 raft_consensus.cc:348] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.668378 11137 raft_consensus.cc:374] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:26.668428 11137 raft_consensus.cc:729] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b26f666e4264ce0b2067919dd736ac6, State: Initialized, Role: FOLLOWER
I20260620 00:13:26.668568 11137 consensus_queue.cc:260] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.668849 11137 ts_tablet_manager.cc:1428] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:13:26.668973 10851 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
I20260620 00:13:26.669018 11140 tablet_bootstrap.cc:492] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: No bootstrap required, opened a new log
I20260620 00:13:26.669070 11140 ts_tablet_manager.cc:1397] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:26.670725 11140 raft_consensus.cc:348] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.670853 11140 raft_consensus.cc:374] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:13:26.670874 11140 raft_consensus.cc:729] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b6a528473e465a9a21045ea63d96d0, State: Initialized, Role: FOLLOWER
I20260620 00:13:26.670984 11140 consensus_queue.cc:260] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.671253 11140 ts_tablet_manager.cc:1428] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:13:26.671288 10985 heartbeater.cc:503] Master 127.0.98.62:36889 was elected leader, sending a full tablet report...
W20260620 00:13:26.710978 10986 tablet.cc:2369] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:13:26.731925 11141 raft_consensus.cc:482] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:13:26.732074 11141 raft_consensus.cc:504] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.732607 11141 leader_election.cc:288] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b26f666e4264ce0b2067919dd736ac6 (127.0.98.1:44839), 49b6a528473e465a9a21045ea63d96d0 (127.0.98.2:45021)
I20260620 00:13:26.735512 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.732827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36689 (local address 127.0.98.1:44839)
0620 00:13:26.732899 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.732901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.732913 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:26.732971 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.732972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.732984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.733223 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:26.733259 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.733852 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.734772 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.735153 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.735238 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.735248 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.735414 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.735449 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:13:26.735514 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.732780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44839 (local address 127.0.98.3:36689)
0620 00:13:26.732851 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.732864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.732906 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.733073 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.733074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.733077 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:26.733158 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.733160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.733936 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.733937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.734673 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.734675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.735239 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.735240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.735323 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.735332 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.735396 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.735439 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 00:13:26.735674 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.732920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45021 (local address 127.0.98.3:41389)
0620 00:13:26.733089 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.733099 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:13:26.733154 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:13:26.733355 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:13:26.733356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:13:26.733358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:13:26.733419 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.733420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.734008 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.734010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.734772 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:13:26.734774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.735350 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.735352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:13:26.735438 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.735444 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:13:26.735519 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:13:26.735556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":15,"thread_start_us":113,"threads_started":1}
I20260620 00:13:26.735922 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5592617e5b841de9bc48c4ee2c37dbe" candidate_uuid: "c923ee04eddb48b8a3a821ae82101274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b26f666e4264ce0b2067919dd736ac6" is_pre_election: true
I20260620 00:13:26.736018 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.732990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41389 (local address 127.0.98.2:45021)
0620 00:13:26.733046 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.733048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.733146 (+ 98us) server_negotiation.cc:407] Connection header received
0620 00:13:26.733228 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.733229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.733276 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.733322 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:13:26.733487 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.733920 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.734873 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.735264 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.735350 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.735359 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.735527 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.735961 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:13:26.736096 10805 raft_consensus.cc:2454] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c923ee04eddb48b8a3a821ae82101274 in term 0.
I20260620 00:13:26.736316 11007 leader_election.cc:302] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b26f666e4264ce0b2067919dd736ac6, c923ee04eddb48b8a3a821ae82101274; no voters:
I20260620 00:13:26.736274 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5592617e5b841de9bc48c4ee2c37dbe" candidate_uuid: "c923ee04eddb48b8a3a821ae82101274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b6a528473e465a9a21045ea63d96d0" is_pre_election: true
I20260620 00:13:26.736403 10939 raft_consensus.cc:2454] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c923ee04eddb48b8a3a821ae82101274 in term 0.
I20260620 00:13:26.736428 11141 raft_consensus.cc:2781] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:13:26.736477 11141 raft_consensus.cc:482] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:13:26.736505 11141 raft_consensus.cc:3037] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:26.737231 11141 raft_consensus.cc:504] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.737370 11141 leader_election.cc:288] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [CANDIDATE]: Term 1 election: Requested vote from peers 0b26f666e4264ce0b2067919dd736ac6 (127.0.98.1:44839), 49b6a528473e465a9a21045ea63d96d0 (127.0.98.2:45021)
I20260620 00:13:26.737552 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5592617e5b841de9bc48c4ee2c37dbe" candidate_uuid: "c923ee04eddb48b8a3a821ae82101274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b6a528473e465a9a21045ea63d96d0"
I20260620 00:13:26.737555 10805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5592617e5b841de9bc48c4ee2c37dbe" candidate_uuid: "c923ee04eddb48b8a3a821ae82101274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b26f666e4264ce0b2067919dd736ac6"
I20260620 00:13:26.737619 10939 raft_consensus.cc:3037] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:26.737641 10805 raft_consensus.cc:3037] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:13:26.738349 10805 raft_consensus.cc:2454] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c923ee04eddb48b8a3a821ae82101274 in term 1.
I20260620 00:13:26.738418 10939 raft_consensus.cc:2454] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c923ee04eddb48b8a3a821ae82101274 in term 1.
I20260620 00:13:26.738628 11007 leader_election.cc:302] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b6a528473e465a9a21045ea63d96d0, c923ee04eddb48b8a3a821ae82101274; no voters:
I20260620 00:13:26.738749 11141 raft_consensus.cc:2781] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:13:26.738950 11141 raft_consensus.cc:686] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 1 LEADER]: Becoming Leader. State: Replica: c923ee04eddb48b8a3a821ae82101274, State: Running, Role: LEADER
I20260620 00:13:26.739053 11141 consensus_queue.cc:237] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } }
I20260620 00:13:26.739885 10659 catalog_manager.cc:5495] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 reported cstate change: term changed from 0 to 1, leader changed from <none> to c923ee04eddb48b8a3a821ae82101274 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c923ee04eddb48b8a3a821ae82101274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c923ee04eddb48b8a3a821ae82101274" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 } health_report { overall_health: UNKNOWN } } }
I20260620 00:13:26.755659 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.753289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.98.3:43519)
0620 00:13:26.753349 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:13:26.753353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:13:26.753365 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:13:26.753395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:13:26.753396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:13:26.753412 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:13:26.753470 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:13:26.753661 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.754115 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.754886 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:13:26.755066 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:13:26.755119 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:13:26.755291 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:13:26.755380 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:13:26.755438 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:13:26.755515 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:13:26.755532 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:13:26.757520 10939 raft_consensus.cc:1261] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 1 FOLLOWER]: Refusing update from remote peer c923ee04eddb48b8a3a821ae82101274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:26.757519 10805 raft_consensus.cc:1261] T f5592617e5b841de9bc48c4ee2c37dbe P 0b26f666e4264ce0b2067919dd736ac6 [term 1 FOLLOWER]: Refusing update from remote peer c923ee04eddb48b8a3a821ae82101274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:13:26.757913 11146 consensus_queue.cc:1035] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b26f666e4264ce0b2067919dd736ac6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:26.758031 11141 consensus_queue.cc:1035] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b6a528473e465a9a21045ea63d96d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:13:26.762657 10659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:26.762764 10659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:13:26.763504 10659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7134eb90239e4c5c9e754a7a9c6f57e4: Sending DeleteTablet for 3 replicas of tablet f5592617e5b841de9bc48c4ee2c37dbe
I20260620 00:13:26.763828 10785 tablet_service.cc:1514] Processing DeleteTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33678
I20260620 00:13:26.763841 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55754
I20260620 00:13:26.763881 10919 tablet_service.cc:1514] Processing DeleteTablet for tablet f5592617e5b841de9bc48c4ee2c37dbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45642
I20260620 00:13:26.764088 11159 tablet_replica.cc:330] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: stopping tablet replica
I20260620 00:13:26.764204 11159 raft_consensus.cc:2229] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:13:26.764281 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10722
I20260620 00:13:26.764303 11159 raft_consensus.cc:2258] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:26.765309 11159 ts_tablet_manager.cc:1905] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:26.766009 11161 tablet_replica.cc:330] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: stopping tablet replica
I20260620 00:13:26.766117 11161 raft_consensus.cc:2229] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:13:26.766224 11161 raft_consensus.cc:2258] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:13:26.766686 11161 ts_tablet_manager.cc:1905] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:13:26.767079 11159 ts_tablet_manager.cc:1918] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:26.767155 11159 log.cc:1196] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/f5592617e5b841de9bc48c4ee2c37dbe
I20260620 00:13:26.767449 11159 ts_tablet_manager.cc:1939] T f5592617e5b841de9bc48c4ee2c37dbe P c923ee04eddb48b8a3a821ae82101274: Deleting consensus metadata
I20260620 00:13:26.768116 10647 catalog_manager.cc:4841] TS c923ee04eddb48b8a3a821ae82101274 (127.0.98.3:43519): tablet f5592617e5b841de9bc48c4ee2c37dbe (table test-table [id=c9464c1800ea4065b95315cb1e133c5d]) successfully deleted
I20260620 00:13:26.768764 11161 ts_tablet_manager.cc:1918] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:13:26.768862 11161 log.cc:1196] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/f5592617e5b841de9bc48c4ee2c37dbe
I20260620 00:13:26.769099 11161 ts_tablet_manager.cc:1939] T f5592617e5b841de9bc48c4ee2c37dbe P 49b6a528473e465a9a21045ea63d96d0: Deleting consensus metadata
I20260620 00:13:26.769564 10645 catalog_manager.cc:4841] TS 49b6a528473e465a9a21045ea63d96d0 (127.0.98.2:45021): tablet f5592617e5b841de9bc48c4ee2c37dbe (table test-table [id=c9464c1800ea4065b95315cb1e133c5d]) successfully deleted
W20260620 00:13:26.770359 10645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44839 (error 108)
I20260620 00:13:26.770614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10856
I20260620 00:13:26.771322 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:13:26.770791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44839 (local address 127.0.0.1:33692)
0620 00:13:26.771245 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 00:13:26.771283 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260620 00:13:26.771732 10645 catalog_manager.cc:4584] TS 0b26f666e4264ce0b2067919dd736ac6 (127.0.98.1:44839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5592617e5b841de9bc48c4ee2c37dbe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44839: connect: Connection refused (error 111)
I20260620 00:13:26.776408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10990
I20260620 00:13:26.781451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 10628
2026-06-20T00:13:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1962 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[11165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[11165](info): set up 2 sockets
Jun 20 00:13:26 dist-test-slave-28bv krb5kdc[11165](info): commencing operation
krb5kdc: starting...
W20260620 00:13:27.820853 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:13:27 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:13:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:13:27.988058 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUZKDhK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:13:30.647 UTC [11195] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:13:30.647 UTC [11195] LOG: listening on IPv4 address "127.0.98.20", port 45193
2026-06-20 00:13:30.651 UTC [11195] LOG: listening on Unix socket "/tmp/.s.PGSQL.45193"
2026-06-20 00:13:30.672 UTC [11198] LOG: database system was shut down at 2026-06-20 00:13:29 UTC
2026-06-20 00:13:30.676 UTC [11195] LOG: database system is ready to accept connections
I20260620 00:13:31.660560 392 mini_postgres.cc:96] Postgres bound to 45193
2026-06-20 00:13:31.666 UTC [11206] FATAL: database "slave" does not exist
127.0.98.20:45193 - accepting connections
I20260620 00:13:31.667129 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:13:31.681959 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:13:32.238715 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:13:32.238809 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin
2026-06-20 00:13:32,539 [I] DB FLAVOR :POSTGRES
2026-06-20 00:13:32,539 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:13:32,540 [I] Checking connection..
2026-06-20 00:13:32,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:13:32,799 [I] Checking connection passed.
2026-06-20 00:13:32,800 [I] --------- Verifying version history table ---------
2026-06-20 00:13:32,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:13:33,052 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:13:33,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:13:33,309 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:13:33,310 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:13:33,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:13:33.546 UTC [11270] WARNING: there is no transaction in progress
2026-06-20 00:13:33,553 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:13:33,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:13:33,804 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:13:33,805 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:13:33,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:13:34,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:13:34,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-28bv.c.gcp-upstream.internal','N') ;"
2026-06-20 00:13:34,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:13:34,831 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:13:34,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:13:35,101 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:13:35,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:13:35,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:13:35,608 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:13:35,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:13:36.675 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.693 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.713 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.731 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.751 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.773 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.843 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.851 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.867 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.878 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.890 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.903 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.913 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.919 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.926 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:36.933 UTC [11378] WARNING: there is no transaction in progress
2026-06-20 00:13:37,666 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:13:37,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:13:37,902 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:13:37,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:13:38,151 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:13:38,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:13:38,426 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:13:38,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:13:38,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:13:38,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-28bv.c.gcp-upstream.internal';"
2026-06-20 00:13:39,158 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:13:39,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:13:39,391 [I] DB_PATCHES have already been applied
I20260620 00:13:39.397832 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUZKDhK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:13:39.397899 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:13:39.401175 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39033
Jun 20, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:13:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:13:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:13:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:13:40.514539 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.847s user 0.002s sys 0.007s
Jun 20, 2026 12:14:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:14:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:14:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:14:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:14:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:14:02.086284 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:14:02 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914442, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:14:02.191437 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:14:02.201597 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:14:03.326480 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:14:03.326573 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms
I20260620 00:14:03.326598 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45193
2026-06-20 00:14:03,527 [I] DB FLAVOR :POSTGRES
2026-06-20 00:14:03,528 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:14:03,528 [I] Checking connection
2026-06-20 00:14:03,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:14:03,801 [I] connection success
2026-06-20 00:14:03,801 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:14:03,801 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:14:03,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:14:04,047 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:14:04,047 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:14:04,047 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:14:04,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45193/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:14:04,315 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:14:04.784761 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:14:04.980594 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:14:05.026497 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:14:05 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914442, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:05.461127 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:14:05.461253 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:14:05.461277 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:14:05.464666 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11558
I20260620 00:14:05.464799 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51061
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:14:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 00:14:07 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:07.852625 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.661s user 0.000s sys 0.184s
I20260620 00:14:07.852743 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:14:07.852828 392 mini_ranger_kms.cc:401] 127.0.98.20:51061/kms/v1/keys
Jun 20 00:14:08 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:14:08 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914447, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:14:10 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:14:10.952693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:14:11.037869 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:11.038085 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:11.038106 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:11.039201 11609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:14:11.039237 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:11.039251 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:11.039263 11609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:14:11.039275 11609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:14:11.040776 11609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:11.040970 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:11.041138 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:11.046348 11617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:11.046452 11615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:11.046499 11609 server_base.cc:1034] running on GCE node
W20260620 00:14:11.046458 11614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:11.046862 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:11.047235 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:11.048384 11609 hybrid_clock.cc:648] HybridClock initialized: now 1781914451048364 us; error 37 us; skew 500 ppm
Jun 20 00:14:11 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:11.051146 11609 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:14:11.051589 11609 webserver.cc:460] Webserver started at http://127.0.98.62:44835/ using document root <none> and password file <none>
I20260620 00:14:11.051793 11609 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:11.051833 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:11.051931 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:11 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:11.097076 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "989b678d6ba1483fae2238762c891bbc"
format_stamp: "Formatted at 2026-06-20 00:14:11 on dist-test-slave-28bv"
server_key: "a91c13f3c45eb63017bdc8fb04936783"
server_key_iv: "65dd2050c29824a3fdbbfa6291ab678d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:11.097506 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "989b678d6ba1483fae2238762c891bbc"
format_stamp: "Formatted at 2026-06-20 00:14:11 on dist-test-slave-28bv"
server_key: "a91c13f3c45eb63017bdc8fb04936783"
server_key_iv: "65dd2050c29824a3fdbbfa6291ab678d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:11.098986 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:14:11.114499 11624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:11.114780 11609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:14:11.114902 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "989b678d6ba1483fae2238762c891bbc"
format_stamp: "Formatted at 2026-06-20 00:14:11 on dist-test-slave-28bv"
server_key: "a91c13f3c45eb63017bdc8fb04936783"
server_key_iv: "65dd2050c29824a3fdbbfa6291ab678d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:11.115010 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:11.147959 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:11.150368 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:11.150547 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:11.156445 11609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260620 00:14:11.156462 11676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260620 00:14:11.156948 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:14:11.157692 11677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:14:11.159546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 11609
I20260620 00:14:11.159667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:14:11.159895 392 external_mini_cluster.cc:1428] Setting key 833639d9ee749c1a3d97e2d12eb94da9
I20260620 00:14:11.161631 11677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: Bootstrap starting.
I20260620 00:14:11.162514 11677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: Neither blocks nor log segments found. Creating new log.
I20260620 00:14:11.163003 11677 log.cc:824] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: Log is configured to *not* fsync() on all Append() calls
I20260620 00:14:11.163929 11677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: No bootstrap required, opened a new log
I20260620 00:14:11.166536 11677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260620 00:14:11.166795 11677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:14:11.166903 11677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989b678d6ba1483fae2238762c891bbc, State: Initialized, Role: FOLLOWER
I20260620 00:14:11.167086 11677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260620 00:14:11.167198 11677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:14:11.167260 11677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:14:11.167340 11677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:14:11.168049 11677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260620 00:14:11.168179 11677 leader_election.cc:302] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 989b678d6ba1483fae2238762c891bbc; no voters:
I20260620 00:14:11.168375 11677 leader_election.cc:288] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:14:11.168431 11682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:14:11.168723 11682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [term 1 LEADER]: Becoming Leader. State: Replica: 989b678d6ba1483fae2238762c891bbc, State: Running, Role: LEADER
Jun 20 00:14:11 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:14:11.168825 11677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:14:11.169379 11682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260620 00:14:11.169994 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "989b678d6ba1483fae2238762c891bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260620 00:14:11.170071 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [sys.catalog]: This master's current role is: LEADER
I20260620 00:14:11.170178 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 989b678d6ba1483fae2238762c891bbc. Latest consensus state: current_term: 1 leader_uuid: "989b678d6ba1483fae2238762c891bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "989b678d6ba1483fae2238762c891bbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260620 00:14:11.170219 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc [sys.catalog]: This master's current role is: LEADER
I20260620 00:14:11.170500 11692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:14:11.170753 11677 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:14:11.171026 11692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:14:11.177507 11680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:11.160510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.98.62:37011)
0620 00:14:11.160749 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:11.160760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:11.160796 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:14:11.161530 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:11.161541 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:11.161621 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:11.161792 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:11.162714 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:11.164209 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:11.165367 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:11.166070 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:11.166203 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:11.169364 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:11.169380 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:11.169380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:11.169393 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:11.174146 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:11.175032 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:11.175041 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:11.175041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:11.175326 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:11.175667 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:11.175668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:11.175669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:11.176353 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:11.176701 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:11.176921 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:14:11.177139 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":129,"threads_started":1}
I20260620 00:14:11.177942 11699 mvcc.cc:205] Tried to move back new op lower bound from 7298721592014135296 to 7298721591990435840. Current Snapshot: MvccSnapshot[applied={T|T < 7298721592014135296}]
I20260620 00:14:11.189244 11692 catalog_manager.cc:1347] Generated new cluster ID: 772ded5b5ea34e64b3a380c090cac3a5
I20260620 00:14:11.189316 11692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:14:11.205124 11692 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:14:11.205708 11692 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:14:11.216205 11692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: Generated new TSK 0
I20260620 00:14:11.216423 11692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:14:11.911646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:14:12.036043 11722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:12.036358 11722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:12.036399 11722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:12.038185 11722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:12.038254 11722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:12.038296 11722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:14:12.040514 11722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:12.040804 11722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:12.041083 11722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:12.046937 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:12.047056 11722 server_base.cc:1034] running on GCE node
W20260620 00:14:12.046955 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:12.046926 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:12.047433 11722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:12.047719 11722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:12.048907 11722 hybrid_clock.cc:648] HybridClock initialized: now 1781914452048891 us; error 27 us; skew 500 ppm
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:12.051946 11722 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:14:12.052541 11722 webserver.cc:460] Webserver started at http://127.0.98.1:37705/ using document root <none> and password file <none>
I20260620 00:14:12.052834 11722 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:12.052911 11722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:12.053051 11722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:12.067637 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd53cf3b271f4c05aced09d692749513"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "f1a338ef8acf6cc7d744db6ad9843f28"
server_key_iv: "64be15db1d54e1c58cdd006ddd28601d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.068224 11722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd53cf3b271f4c05aced09d692749513"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "f1a338ef8acf6cc7d744db6ad9843f28"
server_key_iv: "64be15db1d54e1c58cdd006ddd28601d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.070078 11722 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:14:12.084708 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.085004 11722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:14:12.085115 11722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "cd53cf3b271f4c05aced09d692749513"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "f1a338ef8acf6cc7d744db6ad9843f28"
server_key_iv: "64be15db1d54e1c58cdd006ddd28601d"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.085227 11722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:12.116876 11722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:12.119645 11722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:12.119912 11722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:12.120216 11722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:14:12.120672 11722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:14:12.120771 11722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.120885 11722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:14:12.120963 11722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.129118 11722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43927
I20260620 00:14:12.129523 11722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:14:12.130059 11852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43927 every 8 connection(s)
I20260620 00:14:12.131742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 11722
I20260620 00:14:12.131835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:14:12.132045 392 external_mini_cluster.cc:1428] Setting key db8912c5a0e546edfd6ef140f3ae1502
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:14:12.161682 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.138604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.1:50805)
0620 00:14:12.140751 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.140809 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.141472 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.144999 (+ 3527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.145004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.145280 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:14:12.145570 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.145577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.146700 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.146701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.147860 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.147863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.148271 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.148272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.148399 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.149006 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:14:12.149015 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:14:12.157748 (+ 8733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:14:12.159939 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.159950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.159952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.160281 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.160517 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.160518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.160518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.160683 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.160979 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:14:12.160986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:14:12.161163 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.161271 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.161434 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"thread_start_us":112,"threads_started":1}
I20260620 00:14:12.162437 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.138965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50805 (local address 127.0.98.62:37011)
0620 00:14:12.144683 (+ 5718us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.144686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.144697 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:14:12.144726 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.144727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.144741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.144823 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:12.145713 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.146594 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.147958 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.148167 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.148261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.157894 (+ 9633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:12.157896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:12.157897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:12.157906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:12.159825 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.160370 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.160372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.160372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.160436 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.160754 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.160755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.160755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.160898 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:12.160955 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.162249 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.162341 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5679,"thread_start_us":84,"threads_started":1}
I20260620 00:14:12.163115 11853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260620 00:14:12.163208 11853 heartbeater.cc:464] Registering TS with master...
I20260620 00:14:12.163391 11853 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260620 00:14:12.163966 11641 ts_manager.cc:194] Registered new tserver with Master: cd53cf3b271f4c05aced09d692749513 (127.0.98.1:43927)
I20260620 00:14:12.164983 11641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:14:12.188108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 00:14:12.336424 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:12.336768 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:12.336807 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:12.338564 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:12.338624 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:12.338663 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:14:12.341149 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:12.341444 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:12.341728 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:12.348359 11867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:12.348668 11869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:12.348896 11866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:12.349994 11859 server_base.cc:1034] running on GCE node
I20260620 00:14:12.350247 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:12.350555 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:12.351742 11859 hybrid_clock.cc:648] HybridClock initialized: now 1781914452351715 us; error 36 us; skew 500 ppm
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:12.355312 11859 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:14:12.356060 11859 webserver.cc:460] Webserver started at http://127.0.98.2:37871/ using document root <none> and password file <none>
I20260620 00:14:12.356401 11859 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:12.356567 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:12.356801 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:12.377668 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea289249758348b1b6cae1714a19d01e"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "7071e293251e0b581235c57089cbe963"
server_key_iv: "4263c9832d80270b8f29a21c59681380"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.381917 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea289249758348b1b6cae1714a19d01e"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "7071e293251e0b581235c57089cbe963"
server_key_iv: "4263c9832d80270b8f29a21c59681380"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.384630 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 00:14:12.400736 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.400969 11859 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 00:14:12.401069 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "ea289249758348b1b6cae1714a19d01e"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "7071e293251e0b581235c57089cbe963"
server_key_iv: "4263c9832d80270b8f29a21c59681380"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.401185 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:12.426950 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:12.429363 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:12.429522 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:12.429742 11859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:14:12.430124 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:14:12.430171 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.430209 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:14:12.430236 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.439064 11859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42765
I20260620 00:14:12.439539 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:14:12.439985 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42765 every 8 connection(s)
I20260620 00:14:12.445554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 11859
I20260620 00:14:12.445685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:14:12.445890 392 external_mini_cluster.cc:1428] Setting key 5a5bc8b90f342172381fef5aa3e1c349
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:14:12.459631 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.441162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.2:56419)
0620 00:14:12.441546 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.441591 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.442230 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.442597 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.442601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.442849 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:14:12.443148 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.443154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.444248 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.444249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.445560 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.445564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.446158 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.446159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.446309 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.447505 (+ 1196us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:14:12.447517 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:14:12.449247 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:14:12.457750 (+ 8503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.457765 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.457768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.458140 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.458373 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.458374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.458374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.458535 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.458831 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:14:12.458839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:14:12.459067 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.459189 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.459373 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":96,"threads_started":1}
I20260620 00:14:12.460354 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.441449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56419 (local address 127.0.98.62:37011)
0620 00:14:12.441512 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.441514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.441674 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:14:12.442358 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.442360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.442373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.442405 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:12.443261 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.444032 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.445678 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.446001 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.446052 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.455763 (+ 9711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:12.455766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:12.455766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:12.455778 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:12.457630 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.458242 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.458243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.458244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.458307 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.458603 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.458604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.458604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.458747 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:12.458804 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.460165 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.460264 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:14:12.461189 11990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260620 00:14:12.461288 11990 heartbeater.cc:464] Registering TS with master...
I20260620 00:14:12.461485 11990 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260620 00:14:12.461833 11641 ts_manager.cc:194] Registered new tserver with Master: ea289249758348b1b6cae1714a19d01e (127.0.98.2:42765)
I20260620 00:14:12.462353 11641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:14:12.489324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:14:12.613696 11996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:12.614073 11996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:12.614154 11996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:12.615832 11996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:12.615943 11996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:12.616035 11996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:14:12.618125 11996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:12.618587 11996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:12.618933 11996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:12.626158 12004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:12.626351 12001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:12.627375 11996 server_base.cc:1034] running on GCE node
W20260620 00:14:12.632761 12002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:12.633196 11996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:12.633527 11996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:12.636775 11996 hybrid_clock.cc:648] HybridClock initialized: now 1781914452636750 us; error 40 us; skew 500 ppm
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:12.640211 11996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:14:12.640789 11996 webserver.cc:460] Webserver started at http://127.0.98.3:42249/ using document root <none> and password file <none>
I20260620 00:14:12.641037 11996 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:12.641085 11996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:12.641203 11996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:12.655198 11996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "f54da76840ab482a9e7c1a1186e240cf"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "aea81c5cafcd8dd059fba0aa440bb6b4"
server_key_iv: "0689ce3c1e8f660c4eecc8ded2911986"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.655668 11996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f54da76840ab482a9e7c1a1186e240cf"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "aea81c5cafcd8dd059fba0aa440bb6b4"
server_key_iv: "0689ce3c1e8f660c4eecc8ded2911986"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.662318 11996 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 00:14:12.692040 12011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.692242 11996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:14:12.692337 11996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "f54da76840ab482a9e7c1a1186e240cf"
format_stamp: "Formatted at 2026-06-20 00:14:12 on dist-test-slave-28bv"
server_key: "aea81c5cafcd8dd059fba0aa440bb6b4"
server_key_iv: "0689ce3c1e8f660c4eecc8ded2911986"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:12.692440 11996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:12.715487 11996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:12.718129 11996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:12.718289 11996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:12.718538 11996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:14:12.718914 11996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:14:12.718967 11996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.719015 11996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:14:12.719035 11996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:12.725807 11996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44703
I20260620 00:14:12.726257 11996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:14:12.731179 12124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44703 every 8 connection(s)
I20260620 00:14:12.735193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 11996
I20260620 00:14:12.735311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:14:12.735509 392 external_mini_cluster.cc:1428] Setting key 8482367685e7a7fa73d18a806e219c9e
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:14:12.746676 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.732499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53763 (local address 127.0.98.62:37011)
0620 00:14:12.732573 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.732576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.732820 (+ 244us) server_negotiation.cc:407] Connection header received
0620 00:14:12.733619 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.733621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.733848 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.733888 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:12.734758 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.735751 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.737536 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.738055 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.738109 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.742463 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:12.742465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:12.742465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:12.742475 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:12.744860 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.745471 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.745472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.745473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.745539 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.745846 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.745847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.745848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.746030 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:12.746089 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.746490 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.746578 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:14:12.747259 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.731926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.98.3:53763)
0620 00:14:12.732662 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.732720 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.733425 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.734097 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.734102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.734358 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:14:12.734642 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.734649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.735862 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.735862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.737422 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.737425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.738208 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.738209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.738357 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.739184 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:14:12.739429 (+ 245us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:14:12.742330 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:14:12.744997 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.745028 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.745032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.745368 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.745614 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:12.745615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:12.745615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:12.745762 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:12.746125 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:14:12.746132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:14:12.746320 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.746821 (+ 501us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.747002 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":129,"threads_started":1}
I20260620 00:14:12.747753 12125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37011
I20260620 00:14:12.747855 12125 heartbeater.cc:464] Registering TS with master...
I20260620 00:14:12.748073 12125 heartbeater.cc:511] Master 127.0.98.62:37011 requested a full tablet report, sending...
I20260620 00:14:12.749131 11641 ts_manager.cc:194] Registered new tserver with Master: f54da76840ab482a9e7c1a1186e240cf (127.0.98.3:44703)
I20260620 00:14:12.749626 11641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53763
I20260620 00:14:12.758497 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:14:12.784919 11857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.760154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.0.98.62:37011)
0620 00:14:12.760206 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.760209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.764850 (+ 4641us) server_negotiation.cc:407] Connection header received
0620 00:14:12.776690 (+ 11840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.776693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.776713 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.776751 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:12.777654 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.778504 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.779621 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.779925 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.779968 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.781716 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:12.781718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:12.781718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:12.781727 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:12.783510 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.783942 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.783944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.783944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.784008 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:12.784234 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:12.784235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:12.784236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:12.784373 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:12.784468 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.784732 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.784817 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:14:12.787504 11641 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:14:12.789971 11641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:14:12.799934 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.792726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44703 (local address 127.0.0.1:57748)
0620 00:14:12.793828 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.793873 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.793943 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.794496 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.794500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.794502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:14:12.794580 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.794582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.796168 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.796169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.798199 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.798202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.799615 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.799616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.799697 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.799713 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.799788 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.799841 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":92,"threads_started":1}
I20260620 00:14:12.800395 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.793279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.98.3:44703)
0620 00:14:12.794159 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.794164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.794175 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:14:12.794216 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.794218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.794234 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.794311 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:14:12.794667 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.796069 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.798319 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.799417 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.799490 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.799578 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.800261 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.800302 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":74,"threads_started":1}
I20260620 00:14:12.801138 12053 tablet_service.cc:1467] Processing CreateTablet for tablet d0829270a18f4ef2b0a031f8111727fa (DEFAULT_TABLE table=test-table [id=1d7951b24cfe447a9ddea4a262a4a7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:14:12.801518 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0829270a18f4ef2b0a031f8111727fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:14:12.801648 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.793446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.0.98.2:42765)
0620 00:14:12.794450 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.794454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.796476 (+ 2022us) server_negotiation.cc:407] Connection header received
0620 00:14:12.796528 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.796531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.796601 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.796691 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:14:12.796953 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.798142 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.799630 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.800950 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.801022 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.801099 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.801410 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.801454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":966,"thread_start_us":66,"threads_started":1}
I20260620 00:14:12.801998 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.792976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42765 (local address 127.0.0.1:43356)
0620 00:14:12.796301 (+ 3325us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.796316 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.796400 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.796797 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.796798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.796801 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:14:12.796864 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.796866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.798246 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.798247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.799498 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.799501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.801138 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.801139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.801216 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.801221 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.801276 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.801926 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3300,"thread_start_us":81,"threads_started":1}
I20260620 00:14:12.802824 11924 tablet_service.cc:1467] Processing CreateTablet for tablet d0829270a18f4ef2b0a031f8111727fa (DEFAULT_TABLE table=test-table [id=1d7951b24cfe447a9ddea4a262a4a7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:14:12.803162 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0829270a18f4ef2b0a031f8111727fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:14:12.807605 12144 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Bootstrap starting.
I20260620 00:14:12.808753 12144 tablet_bootstrap.cc:654] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Neither blocks nor log segments found. Creating new log.
I20260620 00:14:12.809159 12144 log.cc:824] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:14:12.809509 12143 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: Bootstrap starting.
I20260620 00:14:12.809921 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.793654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43927 (local address 127.0.0.1:59882)
0620 00:14:12.800146 (+ 6492us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.800157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.800189 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.805711 (+ 5522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.805712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.805714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:14:12.805759 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.805760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.807065 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.807066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.808612 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.808616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.809682 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.809683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.809746 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.809749 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.809793 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.809841 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6480,"thread_start_us":68,"threads_started":1}
I20260620 00:14:12.810209 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.793739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.1:43927)
0620 00:14:12.804175 (+ 10436us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.804180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.804193 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:14:12.804229 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.804231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.804249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.804344 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:14:12.805858 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.806951 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.808713 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.809507 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.809584 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.809648 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.810094 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.810137 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10395,"thread_start_us":59,"threads_started":1}
I20260620 00:14:12.810891 11787 tablet_service.cc:1467] Processing CreateTablet for tablet d0829270a18f4ef2b0a031f8111727fa (DEFAULT_TABLE table=test-table [id=1d7951b24cfe447a9ddea4a262a4a7ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:14:12.811237 12143 tablet_bootstrap.cc:654] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: Neither blocks nor log segments found. Creating new log.
I20260620 00:14:12.811254 11787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0829270a18f4ef2b0a031f8111727fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:14:12.811616 12143 log.cc:824] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: Log is configured to *not* fsync() on all Append() calls
I20260620 00:14:12.812906 12144 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: No bootstrap required, opened a new log
I20260620 00:14:12.812980 12144 ts_tablet_manager.cc:1397] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 00:14:12.814848 12147 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Bootstrap starting.
I20260620 00:14:12.815164 12144 raft_consensus.cc:348] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.815315 12144 raft_consensus.cc:374] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:14:12.815338 12144 raft_consensus.cc:729] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea289249758348b1b6cae1714a19d01e, State: Initialized, Role: FOLLOWER
I20260620 00:14:12.815559 12144 consensus_queue.cc:260] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.815965 12147 tablet_bootstrap.cc:654] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Neither blocks nor log segments found. Creating new log.
I20260620 00:14:12.816063 12144 ts_tablet_manager.cc:1428] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:14:12.816272 12147 log.cc:824] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Log is configured to *not* fsync() on all Append() calls
I20260620 00:14:12.816679 11990 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260620 00:14:12.817054 12147 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: No bootstrap required, opened a new log
I20260620 00:14:12.817127 12147 ts_tablet_manager.cc:1397] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:14:12.817907 12143 tablet_bootstrap.cc:492] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: No bootstrap required, opened a new log
I20260620 00:14:12.817996 12143 ts_tablet_manager.cc:1397] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 00:14:12.819736 12147 raft_consensus.cc:348] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.819911 12147 raft_consensus.cc:374] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:14:12.819957 12147 raft_consensus.cc:729] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd53cf3b271f4c05aced09d692749513, State: Initialized, Role: FOLLOWER
I20260620 00:14:12.820134 12147 consensus_queue.cc:260] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.820217 12143 raft_consensus.cc:348] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.820422 12147 ts_tablet_manager.cc:1428] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:14:12.820434 12143 raft_consensus.cc:374] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:14:12.820570 11853 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
I20260620 00:14:12.820525 12143 raft_consensus.cc:729] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54da76840ab482a9e7c1a1186e240cf, State: Initialized, Role: FOLLOWER
I20260620 00:14:12.820747 12143 consensus_queue.cc:260] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.821043 12143 ts_tablet_manager.cc:1428] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:14:12.821177 12125 heartbeater.cc:503] Master 127.0.98.62:37011 was elected leader, sending a full tablet report...
W20260620 00:14:12.880848 11854 tablet.cc:2369] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:14:12 dist-test-slave-28bv krb5kdc[11165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:12.928347 12151 raft_consensus.cc:482] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:14:12.928500 12151 raft_consensus.cc:504] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.928947 12151 leader_election.cc:288] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd53cf3b271f4c05aced09d692749513 (127.0.98.1:43927), ea289249758348b1b6cae1714a19d01e (127.0.98.2:42765)
I20260620 00:14:12.932965 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.929179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43927 (local address 127.0.98.3:57635)
0620 00:14:12.929496 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.929508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.929552 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.929802 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.929803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.929805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:14:12.929877 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.929879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.930764 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.930765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.931901 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.931903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.932722 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.932792 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.932795 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.932837 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.932875 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260620 00:14:12.933204 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.929678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53879 (local address 127.0.98.2:42765)
0620 00:14:12.929761 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.929763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.929796 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:14:12.929889 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.929890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.929901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.929931 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:14:12.930168 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.930872 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932215 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.932767 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932817 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.932827 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.933093 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.933137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 00:14:12.933238 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.929530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57635 (local address 127.0.98.1:43927)
0620 00:14:12.929582 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.929584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.929595 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:14:12.929626 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.929627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.929637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.929699 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:14:12.929953 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.930675 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932042 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.932621 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932682 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.932694 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.933138 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.933181 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:14:12.933491 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.929308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42765 (local address 127.0.98.3:53879)
0620 00:14:12.929742 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.929751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:12.929819 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:12.929996 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:12.929997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:12.929998 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:14:12.930088 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.930089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.930995 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.930995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.932109 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:12.932112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.932888 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.932889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:12.932971 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.932980 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 00:14:12.933320 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 00:14:12.933374 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":88,"threads_started":1}
I20260620 00:14:12.933476 11807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829270a18f4ef2b0a031f8111727fa" candidate_uuid: "f54da76840ab482a9e7c1a1186e240cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd53cf3b271f4c05aced09d692749513" is_pre_election: true
I20260620 00:14:12.933617 11807 raft_consensus.cc:2454] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54da76840ab482a9e7c1a1186e240cf in term 0.
I20260620 00:14:12.933808 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829270a18f4ef2b0a031f8111727fa" candidate_uuid: "f54da76840ab482a9e7c1a1186e240cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea289249758348b1b6cae1714a19d01e" is_pre_election: true
I20260620 00:14:12.933960 11942 raft_consensus.cc:2454] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54da76840ab482a9e7c1a1186e240cf in term 0.
I20260620 00:14:12.934123 12014 leader_election.cc:302] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd53cf3b271f4c05aced09d692749513, f54da76840ab482a9e7c1a1186e240cf; no voters:
I20260620 00:14:12.934247 12151 raft_consensus.cc:2781] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:14:12.934284 12151 raft_consensus.cc:482] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:14:12.934303 12151 raft_consensus.cc:3037] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:14:12.935159 12151 raft_consensus.cc:504] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.935293 12151 leader_election.cc:288] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [CANDIDATE]: Term 1 election: Requested vote from peers cd53cf3b271f4c05aced09d692749513 (127.0.98.1:43927), ea289249758348b1b6cae1714a19d01e (127.0.98.2:42765)
I20260620 00:14:12.935503 11807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829270a18f4ef2b0a031f8111727fa" candidate_uuid: "f54da76840ab482a9e7c1a1186e240cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd53cf3b271f4c05aced09d692749513"
I20260620 00:14:12.935492 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0829270a18f4ef2b0a031f8111727fa" candidate_uuid: "f54da76840ab482a9e7c1a1186e240cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea289249758348b1b6cae1714a19d01e"
I20260620 00:14:12.935580 11807 raft_consensus.cc:3037] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:14:12.935583 11942 raft_consensus.cc:3037] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:14:12.936431 11807 raft_consensus.cc:2454] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54da76840ab482a9e7c1a1186e240cf in term 1.
I20260620 00:14:12.936611 12014 leader_election.cc:302] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd53cf3b271f4c05aced09d692749513, f54da76840ab482a9e7c1a1186e240cf; no voters:
I20260620 00:14:12.936681 12151 raft_consensus.cc:2781] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:14:12.936815 12151 raft_consensus.cc:686] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 1 LEADER]: Becoming Leader. State: Replica: f54da76840ab482a9e7c1a1186e240cf, State: Running, Role: LEADER
I20260620 00:14:12.936899 12151 consensus_queue.cc:237] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } }
I20260620 00:14:12.936980 11942 raft_consensus.cc:2454] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54da76840ab482a9e7c1a1186e240cf in term 1.
I20260620 00:14:12.937805 11640 catalog_manager.cc:5495] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf reported cstate change: term changed from 0 to 1, leader changed from <none> to f54da76840ab482a9e7c1a1186e240cf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f54da76840ab482a9e7c1a1186e240cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f54da76840ab482a9e7c1a1186e240cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 } health_report { overall_health: UNKNOWN } } }
W20260620 00:14:12.940585 11991 tablet.cc:2369] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:14:12.951963 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.948762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.0.98.3:44703)
0620 00:14:12.948803 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:12.948805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:12.948814 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:14:12.948907 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:12.948908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:12.948929 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:12.949006 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:14:12.949187 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.949890 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.951036 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:12.951317 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:12.951387 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:12.951586 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:14:12.951715 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:14:12.951796 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:12.951866 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 00:14:12.951888 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:14:12.954068 11942 raft_consensus.cc:1261] T d0829270a18f4ef2b0a031f8111727fa P ea289249758348b1b6cae1714a19d01e [term 1 FOLLOWER]: Refusing update from remote peer f54da76840ab482a9e7c1a1186e240cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:14:12.954157 11807 raft_consensus.cc:1261] T d0829270a18f4ef2b0a031f8111727fa P cd53cf3b271f4c05aced09d692749513 [term 1 FOLLOWER]: Refusing update from remote peer f54da76840ab482a9e7c1a1186e240cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:14:12.954516 12153 consensus_queue.cc:1035] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd53cf3b271f4c05aced09d692749513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:14:12.954663 12151 consensus_queue.cc:1035] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea289249758348b1b6cae1714a19d01e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:14:12.957700 12156 mvcc.cc:205] Tried to move back new op lower bound from 7298721599297204224 to 7298721599230148608. Current Snapshot: MvccSnapshot[applied={T|T < 7298721599297204224}]
I20260620 00:14:12.958242 12157 mvcc.cc:205] Tried to move back new op lower bound from 7298721599297204224 to 7298721599230148608. Current Snapshot: MvccSnapshot[applied={T|T < 7298721599297204224}]
I20260620 00:14:12.960665 12160 mvcc.cc:205] Tried to move back new op lower bound from 7298721599297204224 to 7298721599230148608. Current Snapshot: MvccSnapshot[applied={T|T < 7298721599297204224 or (T in {7298721599297204224})}]
I20260620 00:14:12.963096 11640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:14:12.963195 11640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:14:12.964411 11640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 989b678d6ba1483fae2238762c891bbc: Sending DeleteTablet for 3 replicas of tablet d0829270a18f4ef2b0a031f8111727fa
I20260620 00:14:12.964780 12053 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829270a18f4ef2b0a031f8111727fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:14:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57748
I20260620 00:14:12.964891 11787 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829270a18f4ef2b0a031f8111727fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:14:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59882
I20260620 00:14:12.965106 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet d0829270a18f4ef2b0a031f8111727fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:14:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43356
I20260620 00:14:12.965832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 11722
I20260620 00:14:12.972692 12166 tablet_replica.cc:330] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf: stopping tablet replica
I20260620 00:14:12.972844 12166 raft_consensus.cc:2229] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:14:12.972994 12166 raft_consensus.cc:2258] T d0829270a18f4ef2b0a031f8111727fa P f54da76840ab482a9e7c1a1186e240cf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 00:14:12.974793 11627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43927 (error 108)
I20260620 00:14:12.975431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 11859
I20260620 00:14:12.976749 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.976607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43927 (local address 127.0.0.1:59884)
0620 00:14:12.976666 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.976709 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 00:14:12.976979 11627 catalog_manager.cc:4584] TS cd53cf3b271f4c05aced09d692749513 (127.0.98.1:43927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0829270a18f4ef2b0a031f8111727fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43927: connect: Connection refused (error 111)
I20260620 00:14:12.982811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 11996
I20260620 00:14:12.989030 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:12.988894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42765 (local address 127.0.0.1:43360)
0620 00:14:12.988943 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:12.988982 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:14:12.992859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 11609
I20260620 00:14:12.998373 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:14:13.343360 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:14:13.343531 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:14:13.688719 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:14:13.688 UTC [11195] LOG: received smart shutdown request
2026-06-20 00:14:13.692 UTC [11195] LOG: background worker "logical replication launcher" (PID 11204) exited with exit code 1
2026-06-20 00:14:13.692 UTC [11199] LOG: shutting down
2026-06-20 00:14:13.900 UTC [11195] LOG: database system is shut down
2026-06-20T00:14:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47224 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:14:14 dist-test-slave-28bv krb5kdc[12178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:14:14 dist-test-slave-28bv krb5kdc[12178](info): set up 2 sockets
Jun 20 00:14:14 dist-test-slave-28bv krb5kdc[12178](info): commencing operation
krb5kdc: starting...
W20260620 00:14:15.084750 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:14:15 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:14:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:14:15.261363 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUZKDhK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:14:17.984 UTC [12208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:14:17.984 UTC [12208] LOG: listening on IPv4 address "127.0.98.20", port 60281
2026-06-20 00:14:17.989 UTC [12208] LOG: listening on Unix socket "/tmp/.s.PGSQL.60281"
2026-06-20 00:14:18.028 UTC [12211] LOG: database system was shut down at 2026-06-20 00:14:16 UTC
2026-06-20 00:14:18.033 UTC [12208] LOG: database system is ready to accept connections
I20260620 00:14:19.000432 392 mini_postgres.cc:96] Postgres bound to 60281
2026-06-20 00:14:19.006 UTC [12219] FATAL: database "slave" does not exist
127.0.98.20:60281 - accepting connections
I20260620 00:14:19.007252 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:14:19.023147 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:14:19.595556 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:14:19.595654 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin
2026-06-20 00:14:19,871 [I] DB FLAVOR :POSTGRES
2026-06-20 00:14:19,872 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:14:19,872 [I] Checking connection..
2026-06-20 00:14:19,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:14:20,118 [I] Checking connection passed.
2026-06-20 00:14:20,118 [I] --------- Verifying version history table ---------
2026-06-20 00:14:20,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:14:20,368 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:14:20,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:14:20,637 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:14:20,638 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:14:20,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:14:20.890 UTC [12283] WARNING: there is no transaction in progress
2026-06-20 00:14:20,896 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:14:20,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:14:21,143 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:14:21,143 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:14:21,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:14:21,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:14:21,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-28bv.c.gcp-upstream.internal','N') ;"
2026-06-20 00:14:21,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:14:22,167 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:14:22,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:14:22,417 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:14:22,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:14:22,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:14:22,910 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:14:22,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:14:24.031 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.051 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.071 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.090 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.114 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.137 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.211 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.220 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.240 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.254 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.267 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.286 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.300 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.309 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.319 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:24.329 UTC [12391] WARNING: there is no transaction in progress
2026-06-20 00:14:25,210 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:14:25,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:14:25,463 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:14:25,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:14:25,729 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:14:25,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:14:25,984 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:14:25,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:14:26,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:14:26,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-28bv.c.gcp-upstream.internal';"
2026-06-20 00:14:26,760 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:14:26,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:14:27,016 [I] DB_PATCHES have already been applied
I20260620 00:14:27.022449 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUZKDhK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:14:27.022516 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:14:27.025665 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54161
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:14:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:14:28.168401 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.161s user 0.000s sys 0.009s
Jun 20, 2026 12:14:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:14:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:14:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:14:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:14:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:14:50.217159 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:14:50 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914490, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:14:50.298465 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:14:50.308276 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:14:51.285055 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:14:51.285141 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms
I20260620 00:14:51.285163 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60281
2026-06-20 00:14:51,448 [I] DB FLAVOR :POSTGRES
2026-06-20 00:14:51,449 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:14:51,449 [I] Checking connection
2026-06-20 00:14:51,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:14:51,695 [I] connection success
2026-06-20 00:14:51,695 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:14:51,695 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:14:51,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:14:51,941 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:14:51,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:14:51,942 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:14:51,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60281/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:14:52,207 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:14:52.693347 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:14:52.906626 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:14:52.954536 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:14:52 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914490, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:53.381358 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:14:53.381572 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUZKDhK/build/release/bin/postgresql.jar:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUZKDhK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:14:53.381592 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:14:53.384996 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12571
I20260620 00:14:53.385080 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33587
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:14:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 00:14:55.636211 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.338s user 0.000s sys 0.146s
I20260620 00:14:55.636320 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:14:55.636368 392 mini_ranger_kms.cc:401] 127.0.98.20:33587/kms/v1/keys
Jun 20 00:14:55 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 00:14:56 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:14:56 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914495, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:14:58 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:14:58.807684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44119
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:14:58.890910 12622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:58.891180 12622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:58.891219 12622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:58.892421 12622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:14:58.892493 12622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:58.892519 12622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:58.892568 12622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:14:58.892598 12622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:14:58.894220 12622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33587/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44119
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:58.894500 12622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:58.894735 12622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:58.899699 12628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:58.899706 12627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:58.899703 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:58.900152 12622 server_base.cc:1034] running on GCE node
I20260620 00:14:58.900429 12622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:58.900797 12622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:58.901976 12622 hybrid_clock.cc:648] HybridClock initialized: now 1781914498901949 us; error 39 us; skew 500 ppm
Jun 20 00:14:58 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:58.908032 12622 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:14:58.908624 12622 webserver.cc:460] Webserver started at http://127.0.98.62:36323/ using document root <none> and password file <none>
I20260620 00:14:58.908866 12622 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:58.908906 12622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:58.909040 12622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:58 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:58.957259 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "13bbc0ade73f49ffa025aad22e8aa92d"
format_stamp: "Formatted at 2026-06-20 00:14:58 on dist-test-slave-28bv"
server_key: "369bbc21e9f9fe3889ad1e059919f15f"
server_key_iv: "bb90f63bb102aced717af9f15450f315"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:58.957664 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "13bbc0ade73f49ffa025aad22e8aa92d"
format_stamp: "Formatted at 2026-06-20 00:14:58 on dist-test-slave-28bv"
server_key: "369bbc21e9f9fe3889ad1e059919f15f"
server_key_iv: "bb90f63bb102aced717af9f15450f315"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:58.959122 12622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:14:58.971060 12637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:58.971292 12622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:14:58.971395 12622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "13bbc0ade73f49ffa025aad22e8aa92d"
format_stamp: "Formatted at 2026-06-20 00:14:58 on dist-test-slave-28bv"
server_key: "369bbc21e9f9fe3889ad1e059919f15f"
server_key_iv: "bb90f63bb102aced717af9f15450f315"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:58.971511 12622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:58.993021 12622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:58.995266 12622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:58.995450 12622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:59.000991 12622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44119
I20260620 00:14:59.001056 12689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44119 every 8 connection(s)
I20260620 00:14:59.001454 12622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:14:59.002169 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:14:59.004192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 12622
I20260620 00:14:59.004304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:14:59.004477 392 external_mini_cluster.cc:1428] Setting key 1cb1960bc3d3d412a387342fb333db75
I20260620 00:14:59.004758 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Bootstrap starting.
I20260620 00:14:59.005564 12690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Neither blocks nor log segments found. Creating new log.
I20260620 00:14:59.005913 12690 log.cc:824] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:14:59.006697 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: No bootstrap required, opened a new log
I20260620 00:14:59.009229 12690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:14:59.009447 12690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:14:59.009516 12690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bbc0ade73f49ffa025aad22e8aa92d, State: Initialized, Role: FOLLOWER
I20260620 00:14:59.009699 12690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:14:59.009781 12690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:14:59.009811 12690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:14:59.009873 12690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:14:59.010604 12690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:14:59.010720 12690 leader_election.cc:302] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bbc0ade73f49ffa025aad22e8aa92d; no voters:
I20260620 00:14:59.010934 12690 leader_election.cc:288] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:14:59.011001 12695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:14:59.011234 12695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 LEADER]: Becoming Leader. State: Replica: 13bbc0ade73f49ffa025aad22e8aa92d, State: Running, Role: LEADER
I20260620 00:14:59.011298 12690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:14:59.011379 12695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
Jun 20 00:14:59 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:14:59.011790 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } } }
I20260620 00:14:59.011871 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: This master's current role is: LEADER
I20260620 00:14:59.011991 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bbc0ade73f49ffa025aad22e8aa92d. Latest consensus state: current_term: 1 leader_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } } }
I20260620 00:14:59.012059 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: This master's current role is: LEADER
I20260620 00:14:59.012841 12690 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:14:59.013005 12702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:14:59.013595 12702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:14:59.018642 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:59.005183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.98.62:44119)
0620 00:14:59.005349 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:59.005358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:59.005390 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:14:59.006007 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:59.006016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:59.006172 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:59.006349 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:59.007055 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.007943 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.008853 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.009301 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.009427 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:59.012332 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:59.012337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:59.012337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:59.012351 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:59.014461 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:59.014894 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:59.014897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:59.014898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:59.014995 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:59.015224 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:59.015225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:59.015225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:59.017268 (+ 2043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:59.017704 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:59.017820 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 00:14:59.018298 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260620 00:14:59.021721 12702 catalog_manager.cc:1347] Generated new cluster ID: 87574413cdb44259a0d2c516c58a595f
I20260620 00:14:59.021853 12702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:14:59.040973 12702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:14:59.041889 12702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:14:59.048883 12702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Generated new TSK 0
I20260620 00:14:59.049166 12702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:14:59.751955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:14:59.859645 12735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:14:59.859930 12735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:14:59.859967 12735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:14:59.861682 12735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:14:59.861734 12735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:14:59.861763 12735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:14:59.863382 12735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:14:59.863638 12735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:14:59.863889 12735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:14:59.869005 12740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:14:59.869021 12743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:59.869208 12735 server_base.cc:1034] running on GCE node
W20260620 00:14:59.869012 12741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:14:59.869585 12735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:14:59.869845 12735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:14:59.870999 12735 hybrid_clock.cc:648] HybridClock initialized: now 1781914499870985 us; error 33 us; skew 500 ppm
Jun 20 00:14:59 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:14:59.873638 12735 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:14:59.874154 12735 webserver.cc:460] Webserver started at http://127.0.98.1:35309/ using document root <none> and password file <none>
I20260620 00:14:59.874414 12735 fs_manager.cc:359] Metadata directory not provided
I20260620 00:14:59.874460 12735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:14:59.874550 12735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:14:59 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:14:59.887722 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "8227ceca2473449b900d9ffb285b6065"
format_stamp: "Formatted at 2026-06-20 00:14:59 on dist-test-slave-28bv"
server_key: "26307931d918f0f3d440a0a2293c8187"
server_key_iv: "7bbbd196a2128190b71bd4d3cc31b6fe"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:59.888198 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8227ceca2473449b900d9ffb285b6065"
format_stamp: "Formatted at 2026-06-20 00:14:59 on dist-test-slave-28bv"
server_key: "26307931d918f0f3d440a0a2293c8187"
server_key_iv: "7bbbd196a2128190b71bd4d3cc31b6fe"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:59.889827 12735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:14:59.902333 12752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:59.902701 12735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:14:59.902871 12735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "8227ceca2473449b900d9ffb285b6065"
format_stamp: "Formatted at 2026-06-20 00:14:59 on dist-test-slave-28bv"
server_key: "26307931d918f0f3d440a0a2293c8187"
server_key_iv: "7bbbd196a2128190b71bd4d3cc31b6fe"
server_key_version: "kuduclusterkey@0"
I20260620 00:14:59.903079 12735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:14:59.931368 12735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:14:59.934140 12735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:14:59.934365 12735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:14:59.934666 12735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:14:59.935081 12735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:14:59.935174 12735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:59.935268 12735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:14:59.935338 12735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:14:59.941838 12735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40669
I20260620 00:14:59.942404 12735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:14:59.942687 12865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40669 every 8 connection(s)
I20260620 00:14:59.947950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 12735
I20260620 00:14:59.948040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:14:59.948231 392 external_mini_cluster.cc:1428] Setting key 0c1a531bf332dad9fe6a8a880316abad
Jun 20 00:14:59 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:14:59.954334 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:59.943619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.1:36185)
0620 00:14:59.944131 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 00:14:59.944166 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:14:59.944721 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:14:59.945061 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:14:59.945064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:14:59.945249 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:14:59.945436 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:59.945440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.946166 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.946167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:59.946968 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:14:59.946970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.947387 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.947388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:14:59.947504 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:59.948445 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:14:59.948460 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:14:59.950243 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:14:59.952466 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:59.952472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:59.952475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:59.952845 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:59.953069 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:14:59.953070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:14:59.953070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:14:59.953216 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:14:59.953550 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:14:59.953557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:14:59.953764 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 00:14:59.953874 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:14:59.954046 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":92,"threads_started":1}
I20260620 00:14:59.954842 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:14:59.943716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36185 (local address 127.0.98.62:44119)
0620 00:14:59.943887 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 00:14:59.943890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:14:59.944282 (+ 392us) server_negotiation.cc:407] Connection header received
0620 00:14:59.944836 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:14:59.944837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:14:59.944852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:14:59.944926 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:14:59.945541 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.946067 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.947067 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:14:59.947277 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:14:59.947364 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:14:59.950351 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:14:59.950352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:14:59.950352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:14:59.950360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:14:59.952368 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:59.952931 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:59.952932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:59.952933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:59.953001 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:14:59.953295 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:14:59.953296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:14:59.953296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:14:59.953458 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:14:59.953521 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:14:59.954628 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0620 00:14:59.954726 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260620 00:14:59.955325 12866 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:14:59.955451 12866 heartbeater.cc:464] Registering TS with master...
I20260620 00:14:59.955636 12866 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:14:59.956214 12654 ts_manager.cc:194] Registered new tserver with Master: 8227ceca2473449b900d9ffb285b6065 (127.0.98.1:40669)
I20260620 00:14:59.957312 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36185
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:14:59.979851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 00:15:00.110589 12872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:00.110934 12872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:00.110978 12872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:00.113013 12872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:00.113072 12872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:00.113117 12872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:00.115434 12872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:00.115701 12872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:00.116067 12872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:00.122737 12882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.123008 12879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.124153 12880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:00.124528 12872 server_base.cc:1034] running on GCE node
I20260620 00:15:00.124776 12872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:00.125051 12872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:00.126223 12872 hybrid_clock.cc:648] HybridClock initialized: now 1781914500126198 us; error 36 us; skew 500 ppm
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:00.129820 12872 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:00.130409 12872 webserver.cc:460] Webserver started at http://127.0.98.2:33751/ using document root <none> and password file <none>
I20260620 00:15:00.130667 12872 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:00.130721 12872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:00.130851 12872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:00.148142 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "3be386692c9c4d52b0ebbb2109ff6a7e"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "5d4ec325c9c8e0c01fbb8f4dd5c8363c"
server_key_iv: "ca00dce70e66c60c3f45e31fe98133a0"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.148665 12872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3be386692c9c4d52b0ebbb2109ff6a7e"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "5d4ec325c9c8e0c01fbb8f4dd5c8363c"
server_key_iv: "ca00dce70e66c60c3f45e31fe98133a0"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.155875 12872 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260620 00:15:00.171568 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.171844 12872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 00:15:00.171948 12872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "3be386692c9c4d52b0ebbb2109ff6a7e"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "5d4ec325c9c8e0c01fbb8f4dd5c8363c"
server_key_iv: "ca00dce70e66c60c3f45e31fe98133a0"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.172072 12872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:00.201906 12872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:00.204617 12872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:00.204846 12872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:00.205170 12872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:00.205610 12872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:00.205698 12872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.205777 12872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:00.205837 12872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.212493 12872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37653
I20260620 00:15:00.213243 12872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:15:00.213923 13003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37653 every 8 connection(s)
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:00.223284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 12872
I20260620 00:15:00.223412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:15:00.223666 392 external_mini_cluster.cc:1428] Setting key 7764e90fe3e2caea3591a567ffe21c16
I20260620 00:15:00.224406 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.214738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42337 (local address 127.0.98.62:44119)
0620 00:15:00.214782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:00.214785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:00.214912 (+ 127us) server_negotiation.cc:407] Connection header received
0620 00:15:00.215622 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:00.215623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:00.215637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:00.215684 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:00.216432 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.217066 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.218184 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.218421 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.218503 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.220871 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:00.220872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:00.220873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:00.220880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:00.222537 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.223133 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.223135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.223136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.223214 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.223567 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.223568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.223569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.223733 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:00.224004 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:00.224196 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 00:15:00.224297 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:15:00.224654 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.214519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.2:42337)
0620 00:15:00.214737 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:00.214794 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:00.215498 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:00.215765 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:00.215769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:00.216034 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:00.216334 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.216341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.217172 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.217173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.218089 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.218092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.218533 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.218535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.218657 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.219285 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:00.219294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:00.220764 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:00.222652 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.222662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.222665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.223002 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.223310 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.223311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.223311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.223475 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.223826 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:00.223833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:00.224036 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 00:15:00.224177 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 00:15:00.224361 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":103,"threads_started":1}
I20260620 00:15:00.225340 13004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:15:00.225452 13004 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:00.225654 13004 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:00.226169 12654 ts_manager.cc:194] Registered new tserver with Master: 3be386692c9c4d52b0ebbb2109ff6a7e (127.0.98.2:37653)
I20260620 00:15:00.226692 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42337
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:15:00.255915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:00.370772 13009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:00.371052 13009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:00.371094 13009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:00.372385 13009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:00.372466 13009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:00.372514 13009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:00.374336 13009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:00.374701 13009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:00.374972 13009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:15:00.380884 13009 server_base.cc:1034] running on GCE node
W20260620 00:15:00.380823 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.381126 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.380816 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:00.381476 13009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:00.381790 13009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:00.382963 13009 hybrid_clock.cc:648] HybridClock initialized: now 1781914500382943 us; error 36 us; skew 500 ppm
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:00.385810 13009 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:00.386334 13009 webserver.cc:460] Webserver started at http://127.0.98.3:33163/ using document root <none> and password file <none>
I20260620 00:15:00.386556 13009 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:00.386600 13009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:00.386691 13009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:00.409098 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "83c03dfc963b4b958d79a6f1b2633c84"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "83c1ab4e0d704a5dd88dab8868b9c177"
server_key_iv: "e89a0066d9a0b019670429056a82c653"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.409452 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83c03dfc963b4b958d79a6f1b2633c84"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "83c1ab4e0d704a5dd88dab8868b9c177"
server_key_iv: "e89a0066d9a0b019670429056a82c653"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.410830 13009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:00.426230 13024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.426441 13009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 00:15:00.426551 13009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "83c03dfc963b4b958d79a6f1b2633c84"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "83c1ab4e0d704a5dd88dab8868b9c177"
server_key_iv: "e89a0066d9a0b019670429056a82c653"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.426640 13009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:00.448774 13009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:00.451262 13009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:00.451417 13009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:00.451680 13009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:00.452143 13009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:00.452185 13009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.452214 13009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:00.452227 13009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.463263 13009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39889
I20260620 00:15:00.463647 13009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:15:00.464097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13009
I20260620 00:15:00.464214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:15:00.464448 392 external_mini_cluster.cc:1428] Setting key a9eb8164275a6077f2a781a24293eb5d
I20260620 00:15:00.464780 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39889 every 8 connection(s)
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:00.479547 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.465055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48017 (local address 127.0.98.62:44119)
0620 00:15:00.465103 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:00.465105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:00.465350 (+ 245us) server_negotiation.cc:407] Connection header received
0620 00:15:00.466112 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:00.466113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:00.466124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:00.466193 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:00.466784 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.467579 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.469198 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.469498 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.469549 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.472962 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:00.472964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:00.472965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:00.472976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:00.475315 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.477085 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.477087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.477088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.477183 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.477566 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.477567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.477568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.477757 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:00.477826 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:00.479293 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0620 00:15:00.479424 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:15:00.479789 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.465061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.3:48017)
0620 00:15:00.465215 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:00.465262 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:00.466000 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:00.466262 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:00.466267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:00.466473 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:00.466676 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.466679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.467701 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.467703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.469061 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.469065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.469673 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.469674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.469828 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.470677 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:00.470688 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:00.472605 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:00.475484 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.475497 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.475501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.475994 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.477283 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.477284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.477285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.477484 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.477866 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:00.477873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:00.478141 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 00:15:00.479206 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0620 00:15:00.479457 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260620 00:15:00.481079 13138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:15:00.481258 13138 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:00.481513 13138 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:00.482050 12654 ts_manager.cc:194] Registered new tserver with Master: 83c03dfc963b4b958d79a6f1b2633c84 (127.0.98.3:39889)
I20260620 00:15:00.482659 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48017
I20260620 00:15:00.488651 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:15:00.489468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 12735
I20260620 00:15:00.496740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 12872
I20260620 00:15:00.503412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13009
I20260620 00:15:00.510447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 12622
I20260620 00:15:00.516290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44119
--webserver_interface=127.0.98.62
--webserver_port=36323
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44119
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:00.600827 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:00.601042 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:00.601065 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:00.602247 13141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:00.602294 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:00.602309 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:00.602321 13141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:00.602334 13141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:00.603991 13141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44119
--ranger_config_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:00.604224 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:00.604465 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:00.609509 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.609560 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.609509 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:00.609787 13141 server_base.cc:1034] running on GCE node
I20260620 00:15:00.610033 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:00.610380 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:00.611591 13141 hybrid_clock.cc:648] HybridClock initialized: now 1781914500611581 us; error 36 us; skew 500 ppm
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:00.614421 13141 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:00.614883 13141 webserver.cc:460] Webserver started at http://127.0.98.62:36323/ using document root <none> and password file <none>
I20260620 00:15:00.615108 13141 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:00.615191 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:00.655678 13141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:00.656677 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.656844 13141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:00.656961 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "13bbc0ade73f49ffa025aad22e8aa92d"
format_stamp: "Formatted at 2026-06-20 00:14:58 on dist-test-slave-28bv"
server_key: "369bbc21e9f9fe3889ad1e059919f15f"
server_key_iv: "bb90f63bb102aced717af9f15450f315"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.657346 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:00.679250 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:00.681459 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:00.681632 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:00.685763 13141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44119
I20260620 00:15:00.685860 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44119 every 8 connection(s)
I20260620 00:15:00.686215 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:00.687472 13210 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:15:00.688145 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Bootstrap starting.
I20260620 00:15:00.693008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13141
I20260620 00:15:00.693374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40669
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44119
--webserver_port=35309
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:00.697230 13210 log.cc:824] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:00.699043 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:00.699358 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Bootstrap complete.
I20260620 00:15:00.701912 13210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:15:00.702152 13210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13bbc0ade73f49ffa025aad22e8aa92d, State: Initialized, Role: FOLLOWER
I20260620 00:15:00.702330 13210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:15:00.702409 13210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:00.702476 13210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:00.702525 13210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:15:00.703600 13210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:15:00.703694 13210 leader_election.cc:302] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13bbc0ade73f49ffa025aad22e8aa92d; no voters:
I20260620 00:15:00.703929 13210 leader_election.cc:288] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:15:00.703970 13214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:15:00.704082 13214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [term 2 LEADER]: Becoming Leader. State: Replica: 13bbc0ade73f49ffa025aad22e8aa92d, State: Running, Role: LEADER
I20260620 00:15:00.704201 13210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:00.704169 13214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } }
I20260620 00:15:00.704468 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } } }
I20260620 00:15:00.704561 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:00.704686 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13bbc0ade73f49ffa025aad22e8aa92d. Latest consensus state: current_term: 2 leader_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13bbc0ade73f49ffa025aad22e8aa92d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44119 } } }
I20260620 00:15:00.704758 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:00.705047 13218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:00.705580 13218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:15:00.705722 13218 catalog_manager.cc:1259] Loaded cluster ID: 87574413cdb44259a0d2c516c58a595f
I20260620 00:15:00.705775 13218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:00.705937 13210 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:15:00.709138 13218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:00.709704 13218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Loaded TSK: 0
I20260620 00:15:00.709992 13218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:15:00.808409 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:00.808719 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:00.808761 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:00.810448 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:00.810508 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:00.810547 13212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:00.812758 13212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35309
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:00.813009 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:00.813289 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:00.820060 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:00.820389 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:00.820574 13212 server_base.cc:1034] running on GCE node
W20260620 00:15:00.820770 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:00.821082 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:00.821417 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:00.822614 13212 hybrid_clock.cc:648] HybridClock initialized: now 1781914500822580 us; error 45 us; skew 500 ppm
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:00.825837 13212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:00.826395 13212 webserver.cc:460] Webserver started at http://127.0.98.1:35309/ using document root <none> and password file <none>
I20260620 00:15:00.826797 13212 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:00.826891 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:00.850056 13212 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 00:15:00.850939 13258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.851161 13212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:00.851274 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "8227ceca2473449b900d9ffb285b6065"
format_stamp: "Formatted at 2026-06-20 00:14:59 on dist-test-slave-28bv"
server_key: "26307931d918f0f3d440a0a2293c8187"
server_key_iv: "7bbbd196a2128190b71bd4d3cc31b6fe"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:00.851764 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:00.868974 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:00.871512 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:00.871727 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:00.872009 13212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:00.872434 13212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:00.872504 13212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.872586 13212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:00.872625 13212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:00.879806 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40669
I20260620 00:15:00.880402 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:15:00.881299 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40669 every 8 connection(s)
I20260620 00:15:00.890311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13212
I20260620 00:15:00.890627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37653
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44119
--webserver_port=33751
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:15:00 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:00.905889 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.885345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.1:42773)
0620 00:15:00.886179 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:00.886227 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:00.892599 (+ 6372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:00.893282 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:00.893286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:00.893537 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:00.893810 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.893816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.895731 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.895733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.897278 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:00.897282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.898419 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.898420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:00.898586 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.899221 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:00.899232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:00.900952 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:00.903462 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.903469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.903472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.903832 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.904200 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:00.904201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:00.904201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:00.904353 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:00.905034 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:00.905041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:00.905231 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 00:15:00.905347 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 00:15:00.905536 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":127,"threads_started":1}
I20260620 00:15:00.907389 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:00.885792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42773 (local address 127.0.98.62:44119)
0620 00:15:00.886061 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:00.886072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:00.887541 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 00:15:00.892935 (+ 5394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:00.892942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:00.893006 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:00.893137 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:00.893962 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.895556 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.897402 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:00.898165 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:00.898288 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:00.901100 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:00.901105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:00.901105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:00.901119 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:00.903346 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.903977 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.903981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.903982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.904109 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:00.904436 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:00.904437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:00.904437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:00.904867 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:00.904966 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:00.907002 (+ 2036us) server_negotiation.cc:299] Negotiation successful
0620 00:15:00.907164 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260620 00:15:00.908362 13372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:15:00.908465 13372 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:00.908790 13372 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:00.909693 13372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:15:00.909765 13372 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:00.909842 13372 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:00.910241 13372 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:00.910315 13372 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:00.910655 13372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:15:01.005405 13376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:01.005720 13376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:01.005760 13376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:01.008190 13376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:01.008275 13376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:01.008322 13376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:01.010982 13376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33751
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:01.011277 13376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:01.011559 13376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:01.018635 13381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:01.018646 13384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:01.018652 13382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:01.019202 13376 server_base.cc:1034] running on GCE node
I20260620 00:15:01.019465 13376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:01.019759 13376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:01.020944 13376 hybrid_clock.cc:648] HybridClock initialized: now 1781914501020923 us; error 35 us; skew 500 ppm
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:01.023842 13376 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:01.024441 13376 webserver.cc:460] Webserver started at http://127.0.98.2:33751/ using document root <none> and password file <none>
I20260620 00:15:01.024716 13376 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:01.024806 13376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:01.042761 13376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:01.043624 13392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.043845 13376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:01.043941 13376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "3be386692c9c4d52b0ebbb2109ff6a7e"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "5d4ec325c9c8e0c01fbb8f4dd5c8363c"
server_key_iv: "ca00dce70e66c60c3f45e31fe98133a0"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:01.044407 13376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:01.078217 13376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:01.081311 13376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:01.081501 13376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:01.081799 13376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:01.082192 13376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:01.082248 13376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.082275 13376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:01.082288 13376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.090013 13376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37653
I20260620 00:15:01.090117 13505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37653 every 8 connection(s)
I20260620 00:15:01.090467 13376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:15:01.098441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13376
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:01.098783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33587/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39889
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44119
--webserver_port=33163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:01.104904 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:01.091459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49795 (local address 127.0.98.62:44119)
0620 00:15:01.091531 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:01.091533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:01.091786 (+ 253us) server_negotiation.cc:407] Connection header received
0620 00:15:01.092558 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:01.092559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:01.092585 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:01.092655 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:01.093422 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.094242 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.095816 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.096132 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.096220 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:01.100628 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:01.100630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:01.100630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:01.100640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:01.102586 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.103286 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.103289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.103289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.103362 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.103759 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.103761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.103761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.103928 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:01.104003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:01.104467 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 00:15:01.104650 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:15:01.104962 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:01.091330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.2:49795)
0620 00:15:01.091607 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:01.091664 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:01.092395 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:01.092738 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:01.092742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:01.093004 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:01.093298 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:01.093305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.094366 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.094368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:01.095704 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:01.095707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.096264 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.096265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:01.096380 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:01.097047 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:01.097059 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:01.100432 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:01.102731 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:01.102744 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:01.102748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:01.103182 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:01.103453 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:01.103454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:01.103454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:01.103648 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:01.104011 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:01.104020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:01.104259 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 00:15:01.104399 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 00:15:01.104630 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260620 00:15:01.105602 13506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:15:01.105737 13506 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.105980 13506 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:01.106786 13506 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:15:01.106890 13506 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.106979 13506 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:01.107597 13506 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.107702 13506 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:01.108151 13506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:15:01.213915 13509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:01.214215 13509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:01.214260 13509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:01.215791 13509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:01.215857 13509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:01.215903 13509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:01.217587 13509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:33587/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33163
--tserver_master_addrs=127.0.98.62:44119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:01.217859 13509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:01.218092 13509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:01.224206 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:01.224313 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:01.224210 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:01.225968 13509 server_base.cc:1034] running on GCE node
I20260620 00:15:01.226233 13509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:01.226555 13509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:01.227751 13509 hybrid_clock.cc:648] HybridClock initialized: now 1781914501227735 us; error 32 us; skew 500 ppm
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:01.231328 13509 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:01.231887 13509 webserver.cc:460] Webserver started at http://127.0.98.3:33163/ using document root <none> and password file <none>
I20260620 00:15:01.232169 13509 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:01.232259 13509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:01.247614 13509 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:15:01.248574 13526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.248833 13509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:01.248941 13509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "83c03dfc963b4b958d79a6f1b2633c84"
format_stamp: "Formatted at 2026-06-20 00:15:00 on dist-test-slave-28bv"
server_key: "83c1ab4e0d704a5dd88dab8868b9c177"
server_key_iv: "e89a0066d9a0b019670429056a82c653"
server_key_version: "kuduclusterkey@0"
I20260620 00:15:01.249439 13509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:01.266403 13509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:01.269018 13509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:01.269174 13509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:01.269459 13509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:01.269827 13509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:01.269878 13509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.269913 13509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:01.269942 13509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:01.276346 13509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39889
I20260620 00:15:01.276594 13639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39889 every 8 connection(s)
I20260620 00:15:01.276844 13509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:01.285084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13509
I20260620 00:15:01.290993 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:01.277605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59299 (local address 127.0.98.62:44119)
0620 00:15:01.277674 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:01.277676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:01.277892 (+ 216us) server_negotiation.cc:407] Connection header received
0620 00:15:01.278628 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:01.278629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:01.278645 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:01.278727 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:01.279467 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.280263 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.281621 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.281945 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.282027 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:01.284253 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:01.284255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:01.284256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:01.284265 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:01.285855 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.286492 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.286494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.286495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.286561 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.290137 (+ 3576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.290139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.290139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.290300 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:01.290351 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:01.290788 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 00:15:01.290891 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:15:01.291592 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:01.277586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44119 (local address 127.0.98.3:59299)
0620 00:15:01.277761 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:01.277805 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:01.278453 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:01.278803 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:01.278807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:01.279057 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:01.279348 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:01.279353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.280363 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.280364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:01.281515 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:01.281518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.282044 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.282045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:01.282143 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:01.282784 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:01.282793 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:01.284146 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:01.285958 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:01.285966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:01.285969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:01.286394 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:01.289834 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:01.289835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:01.289836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:01.290035 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:01.290406 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:01.290414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:01.290617 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 00:15:01.291169 (+ 552us) client_negotiation.cc:240] Negotiation successful
0620 00:15:01.291352 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260620 00:15:01.292171 13640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44119
I20260620 00:15:01.292285 13640 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.292485 13640 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:01.293179 13640 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44119 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:15:01.293265 13640 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.293329 13640 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:01.293665 13640 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.293757 13640 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
W20260620 00:15:01.294057 13640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:15:01.300483 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:01.286310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.0.98.62:44119)
0620 00:15:01.286589 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:01.286591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:01.286599 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:15:01.286616 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:01.286617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:01.286630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:01.286718 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:01.287487 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.288264 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.295052 (+ 6788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:01.295267 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:01.295364 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:01.297748 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:01.297749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:01.297750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:01.297757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:01.299291 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.299676 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.299677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.299678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.299735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:01.299947 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:01.299948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:01.299948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:01.300059 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:01.300242 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:01.300280 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:15:01.300368 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":42,"threads_started":1}
Jun 20 00:15:01 dist-test-slave-28bv krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:01.910848 13372 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:01.911026 13372 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:01.911489 13174 ts_manager.cc:194] Registered new tserver with Master: 8227ceca2473449b900d9ffb285b6065 (127.0.98.1:40669)
I20260620 00:15:01.912827 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42773
I20260620 00:15:02.108345 13506 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:02.108492 13506 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:02.108976 13174 ts_manager.cc:194] Registered new tserver with Master: 3be386692c9c4d52b0ebbb2109ff6a7e (127.0.98.2:37653)
I20260620 00:15:02.109505 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49795
Jun 20 00:15:02 dist-test-slave-28bv krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:15:02.294237 13640 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:02.294396 13640 heartbeater.cc:511] Master 127.0.98.62:44119 requested a full tablet report, sending...
I20260620 00:15:02.294673 13174 ts_manager.cc:194] Registered new tserver with Master: 83c03dfc963b4b958d79a6f1b2633c84 (127.0.98.3:39889)
I20260620 00:15:02.295176 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59299
I20260620 00:15:02.305585 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:15:02.314003 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.306653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.98.62:44119)
0620 00:15:02.306772 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.306774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.306832 (+ 58us) server_negotiation.cc:407] Connection header received
0620 00:15:02.306912 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.306912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.306923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.306977 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:02.307687 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.308207 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.309281 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.309489 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.309557 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.311003 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:02.311005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:02.311005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:02.311012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:02.312465 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:02.312927 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:02.312928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:02.312928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:02.312992 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:02.313257 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:02.313258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:02.313259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:02.313387 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:02.313479 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.313715 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.313798 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260620 00:15:02.316231 13174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:15:02.318174 13174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:15:02.326629 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.0.98.2:37653)
0620 00:15:02.321833 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.321838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.322071 (+ 233us) server_negotiation.cc:407] Connection header received
0620 00:15:02.322118 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.322120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.322136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.322227 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:02.322588 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.323700 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.325402 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.326189 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.326236 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.326317 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.326510 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.326550 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260620 00:15:02.326589 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39889 (local address 127.0.0.1:55580)
0620 00:15:02.321567 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.321609 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:02.321671 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:02.322333 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:02.322337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:02.322342 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:02.322439 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.322442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.323808 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.323809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.325294 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.325298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.326270 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.326271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.326368 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.326383 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:15:02.326430 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:15:02.326497 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":59,"thread_start_us":87,"threads_started":1}
I20260620 00:15:02.326858 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37653 (local address 127.0.0.1:52510)
0620 00:15:02.321988 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.321997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:02.322058 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:02.322333 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:02.322337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:02.322339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:02.322502 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.322505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.323805 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.323806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.325295 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.325298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.326371 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.326372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.326426 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.326429 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:15:02.326716 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 00:15:02.326772 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":174,"thread_start_us":82,"threads_started":1}
I20260620 00:15:02.327834 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 (DEFAULT_TABLE table=test-table [id=73a0186c41f048e59b6b8b95048be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:02.328238 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cff45f3b3ce43e098117f1ec553ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:02.331192 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.0.98.3:39889)
0620 00:15:02.322167 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.322173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.322183 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:15:02.322214 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.322216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.322230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.322521 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:02.322586 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.323708 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.325396 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.326151 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.330898 (+ 4747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.331016 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.331047 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.331107 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":55,"threads_started":1}
I20260620 00:15:02.331784 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40669 (local address 127.0.0.1:33514)
0620 00:15:02.326707 (+ 5035us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.326733 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:02.326779 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:02.327496 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:02.327497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:02.327499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:02.327533 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.327534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.328856 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.328857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.329879 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.329882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.331535 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.331537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.331608 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.331613 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:02.331660 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:15:02.331708 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5022,"thread_start_us":83,"threads_started":1}
I20260620 00:15:02.331954 13574 tablet_service.cc:1467] Processing CreateTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 (DEFAULT_TABLE table=test-table [id=73a0186c41f048e59b6b8b95048be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:02.332211 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.321833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.0.98.1:40669)
0620 00:15:02.321944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.321948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.327326 (+ 5378us) server_negotiation.cc:407] Connection header received
0620 00:15:02.327377 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.327380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.327398 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.327597 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:02.327686 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.328751 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.329993 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.330769 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.331972 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.332066 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.332115 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.332153 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260620 00:15:02.332463 13574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cff45f3b3ce43e098117f1ec553ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:02.332940 13293 tablet_service.cc:1467] Processing CreateTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 (DEFAULT_TABLE table=test-table [id=73a0186c41f048e59b6b8b95048be6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:02.333267 13293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cff45f3b3ce43e098117f1ec553ded4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:02.335873 13674 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Bootstrap starting.
I20260620 00:15:02.336086 13673 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Bootstrap starting.
I20260620 00:15:02.336934 13674 tablet_bootstrap.cc:654] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:02.337136 13673 tablet_bootstrap.cc:654] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:02.337378 13674 log.cc:824] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:02.337432 13673 log.cc:824] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:02.337785 13675 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Bootstrap starting.
I20260620 00:15:02.338238 13673 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: No bootstrap required, opened a new log
I20260620 00:15:02.338713 13673 ts_tablet_manager.cc:1397] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 00:15:02.339026 13675 tablet_bootstrap.cc:654] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:02.339445 13675 log.cc:824] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:02.340818 13674 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: No bootstrap required, opened a new log
I20260620 00:15:02.340934 13674 ts_tablet_manager.cc:1397] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 00:15:02.341892 13673 raft_consensus.cc:348] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.342082 13673 raft_consensus.cc:374] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:02.342126 13673 raft_consensus.cc:729] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be386692c9c4d52b0ebbb2109ff6a7e, State: Initialized, Role: FOLLOWER
I20260620 00:15:02.342281 13673 consensus_queue.cc:260] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.342559 13673 ts_tablet_manager.cc:1428] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:15:02.342701 13506 heartbeater.cc:503] Master 127.0.98.62:44119 was elected leader, sending a full tablet report...
I20260620 00:15:02.344301 13674 raft_consensus.cc:348] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.344518 13674 raft_consensus.cc:374] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:02.344558 13674 raft_consensus.cc:729] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8227ceca2473449b900d9ffb285b6065, State: Initialized, Role: FOLLOWER
I20260620 00:15:02.344722 13674 consensus_queue.cc:260] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.345016 13372 heartbeater.cc:503] Master 127.0.98.62:44119 was elected leader, sending a full tablet report...
I20260620 00:15:02.345291 13674 ts_tablet_manager.cc:1428] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 00:15:02.346227 13675 tablet_bootstrap.cc:492] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: No bootstrap required, opened a new log
I20260620 00:15:02.346330 13675 ts_tablet_manager.cc:1397] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 00:15:02.348409 13675 raft_consensus.cc:348] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.348613 13675 raft_consensus.cc:374] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:02.348654 13675 raft_consensus.cc:729] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c03dfc963b4b958d79a6f1b2633c84, State: Initialized, Role: FOLLOWER
I20260620 00:15:02.348858 13675 consensus_queue.cc:260] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.349568 13675 ts_tablet_manager.cc:1428] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:15:02.349727 13640 heartbeater.cc:503] Master 127.0.98.62:44119 was elected leader, sending a full tablet report...
I20260620 00:15:02.352962 13679 raft_consensus.cc:482] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:02.353183 13679 raft_consensus.cc:504] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.353595 13679 leader_election.cc:288] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c03dfc963b4b958d79a6f1b2633c84 (127.0.98.3:39889), 8227ceca2473449b900d9ffb285b6065 (127.0.98.1:40669)
I20260620 00:15:02.358590 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.354098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58641 (local address 127.0.98.3:39889)
0620 00:15:02.354160 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.354163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.354487 (+ 324us) server_negotiation.cc:407] Connection header received
0620 00:15:02.354612 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.354613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.354626 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.354658 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:02.354949 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.355824 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.357618 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.358217 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.358257 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.358267 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.358481 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.358520 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:15:02.358590 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.354131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52103 (local address 127.0.98.1:40669)
0620 00:15:02.354174 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.354175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.354521 (+ 346us) server_negotiation.cc:407] Connection header received
0620 00:15:02.354632 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.354634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.354647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.354676 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:02.354948 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.355790 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.357533 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.358162 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.358213 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.358227 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.358462 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.358509 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:15:02.358851 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.353780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39889 (local address 127.0.98.2:58641)
0620 00:15:02.354394 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.354407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:02.354517 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:02.354734 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:02.354735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:02.354737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:02.354865 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.354868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.356395 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.356396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.357530 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.357532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.358335 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.358336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.358402 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.358405 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:15:02.358688 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 00:15:02.358741 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":2,"thread_start_us":153,"threads_started":1}
I20260620 00:15:02.359535 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.354233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40669 (local address 127.0.98.2:52103)
0620 00:15:02.354463 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.354472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:02.354544 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:02.354734 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:02.354735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:02.354740 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:02.354865 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.354869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.355919 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.355919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.357428 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:02.357432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.358292 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.358293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:02.358372 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.358377 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:02.359310 (+ 933us) client_negotiation.cc:240] Negotiation successful
0620 00:15:02.359359 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":26,"thread_start_us":107,"threads_started":1}
I20260620 00:15:02.359649 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cff45f3b3ce43e098117f1ec553ded4" candidate_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c03dfc963b4b958d79a6f1b2633c84" is_pre_election: true
I20260620 00:15:02.359856 13594 raft_consensus.cc:2454] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3be386692c9c4d52b0ebbb2109ff6a7e in term 0.
I20260620 00:15:02.359902 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cff45f3b3ce43e098117f1ec553ded4" candidate_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8227ceca2473449b900d9ffb285b6065" is_pre_election: true
I20260620 00:15:02.360069 13322 raft_consensus.cc:2454] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3be386692c9c4d52b0ebbb2109ff6a7e in term 0.
I20260620 00:15:02.360159 13396 leader_election.cc:302] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be386692c9c4d52b0ebbb2109ff6a7e, 83c03dfc963b4b958d79a6f1b2633c84; no voters:
I20260620 00:15:02.360299 13679 raft_consensus.cc:2781] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:02.360355 13679 raft_consensus.cc:482] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:02.360391 13679 raft_consensus.cc:3037] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:02.361415 13679 raft_consensus.cc:504] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.361562 13679 leader_election.cc:288] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [CANDIDATE]: Term 1 election: Requested vote from peers 83c03dfc963b4b958d79a6f1b2633c84 (127.0.98.3:39889), 8227ceca2473449b900d9ffb285b6065 (127.0.98.1:40669)
I20260620 00:15:02.361768 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cff45f3b3ce43e098117f1ec553ded4" candidate_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c03dfc963b4b958d79a6f1b2633c84"
I20260620 00:15:02.361786 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cff45f3b3ce43e098117f1ec553ded4" candidate_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8227ceca2473449b900d9ffb285b6065"
I20260620 00:15:02.361851 13322 raft_consensus.cc:3037] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:02.361851 13594 raft_consensus.cc:3037] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:02.362754 13322 raft_consensus.cc:2454] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be386692c9c4d52b0ebbb2109ff6a7e in term 1.
I20260620 00:15:02.362754 13594 raft_consensus.cc:2454] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3be386692c9c4d52b0ebbb2109ff6a7e in term 1.
I20260620 00:15:02.362946 13396 leader_election.cc:302] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3be386692c9c4d52b0ebbb2109ff6a7e, 83c03dfc963b4b958d79a6f1b2633c84; no voters:
I20260620 00:15:02.363044 13679 raft_consensus.cc:2781] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:02.363209 13679 raft_consensus.cc:686] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 1 LEADER]: Becoming Leader. State: Replica: 3be386692c9c4d52b0ebbb2109ff6a7e, State: Running, Role: LEADER
I20260620 00:15:02.363318 13679 consensus_queue.cc:237] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } }
I20260620 00:15:02.364279 13173 catalog_manager.cc:5495] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3be386692c9c4d52b0ebbb2109ff6a7e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3be386692c9c4d52b0ebbb2109ff6a7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37653 } health_report { overall_health: HEALTHY } } }
I20260620 00:15:02.378721 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.376082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.98.2:37653)
0620 00:15:02.376132 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:02.376134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:02.376182 (+ 48us) server_negotiation.cc:407] Connection header received
0620 00:15:02.376274 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:02.376275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:02.376295 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:02.376378 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:02.376581 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.377072 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.377888 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:02.378087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:02.378177 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:02.378374 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:02.378484 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:02.378582 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:02.378602 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 00:15:02.378627 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:15:02.380714 13594 raft_consensus.cc:1261] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 1 FOLLOWER]: Refusing update from remote peer 3be386692c9c4d52b0ebbb2109ff6a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:02.381275 13322 raft_consensus.cc:1261] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065 [term 1 FOLLOWER]: Refusing update from remote peer 3be386692c9c4d52b0ebbb2109ff6a7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:02.381294 13679 consensus_queue.cc:1035] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c03dfc963b4b958d79a6f1b2633c84" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:02.381752 13679 consensus_queue.cc:1035] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8227ceca2473449b900d9ffb285b6065" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:15:02.381875 13373 tablet.cc:2369] T 4cff45f3b3ce43e098117f1ec553ded4 P 8227ceca2473449b900d9ffb285b6065: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:02.385378 13687 mvcc.cc:205] Tried to move back new op lower bound from 7298721801748287488 to 7298721801680814080. Current Snapshot: MvccSnapshot[applied={T|T < 7298721801748287488}]
I20260620 00:15:02.385752 13688 mvcc.cc:205] Tried to move back new op lower bound from 7298721801748287488 to 7298721801680814080. Current Snapshot: MvccSnapshot[applied={T|T < 7298721801748287488}]
I20260620 00:15:02.390720 13173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:02.390779 13690 mvcc.cc:205] Tried to move back new op lower bound from 7298721801748287488 to 7298721801680814080. Current Snapshot: MvccSnapshot[applied={T|T < 7298721801748287488}]
I20260620 00:15:02.390864 13173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:02.392237 13173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13bbc0ade73f49ffa025aad22e8aa92d: Sending DeleteTablet for 3 replicas of tablet 4cff45f3b3ce43e098117f1ec553ded4
I20260620 00:15:02.392722 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52510
I20260620 00:15:02.393075 13293 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33514
I20260620 00:15:02.393357 13292 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33514
I20260620 00:15:02.393538 13574 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cff45f3b3ce43e098117f1ec553ded4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55580
I20260620 00:15:02.393821 13698 tablet_replica.cc:330] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: stopping tablet replica
I20260620 00:15:02.394001 13698 raft_consensus.cc:2229] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:02.394208 13698 raft_consensus.cc:2258] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:02.394848 13698 ts_tablet_manager.cc:1905] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:02.395344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13212
I20260620 00:15:02.398164 13698 ts_tablet_manager.cc:1918] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:02.398363 13698 log.cc:1196] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/4cff45f3b3ce43e098117f1ec553ded4
I20260620 00:15:02.398763 13698 ts_tablet_manager.cc:1939] T 4cff45f3b3ce43e098117f1ec553ded4 P 83c03dfc963b4b958d79a6f1b2633c84: Deleting consensus metadata
I20260620 00:15:02.399533 13161 catalog_manager.cc:4841] TS 83c03dfc963b4b958d79a6f1b2633c84 (127.0.98.3:39889): tablet 4cff45f3b3ce43e098117f1ec553ded4 (table test-table [id=73a0186c41f048e59b6b8b95048be6cb]) successfully deleted
I20260620 00:15:02.404307 13696 tablet_replica.cc:330] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: stopping tablet replica
I20260620 00:15:02.404621 13696 raft_consensus.cc:2229] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:15:02.404867 13696 raft_consensus.cc:2258] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:02.405699 13696 ts_tablet_manager.cc:1905] T 4cff45f3b3ce43e098117f1ec553ded4 P 3be386692c9c4d52b0ebbb2109ff6a7e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:15:02.406422 13159 connection.cc:535] server connection from 127.0.98.1:42773 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:15:02.406622 13161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40669 (error 108)
I20260620 00:15:02.407478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13376
I20260620 00:15:02.407510 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.407324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40669 (local address 127.0.0.1:33522)
0620 00:15:02.407397 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.407465 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:15:02.407655 13161 catalog_manager.cc:4584] TS 8227ceca2473449b900d9ffb285b6065 (127.0.98.1:40669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4cff45f3b3ce43e098117f1ec553ded4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40669: connect: Connection refused (error 111)
I20260620 00:15:02.409175 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.409056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40669 (local address 127.0.0.1:33534)
0620 00:15:02.409107 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.409133 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:15:02.413059 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:02.412937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37653 (local address 127.0.0.1:52526)
0620 00:15:02.412985 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:02.413009 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 00:15:02.413164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13509
I20260620 00:15:02.418807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13141
I20260620 00:15:02.425020 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:15:02.759831 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:15:02.759958 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:15:03.105298 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:15:03.105 UTC [12208] LOG: received smart shutdown request
2026-06-20 00:15:03.109 UTC [12208] LOG: background worker "logical replication launcher" (PID 12217) exited with exit code 1
2026-06-20 00:15:03.109 UTC [12212] LOG: shutting down
2026-06-20 00:15:03.310 UTC [12208] LOG: database system is shut down
2026-06-20T00:15:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49415 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:15:03 dist-test-slave-28bv krb5kdc[13708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:15:03 dist-test-slave-28bv krb5kdc[13708](info): set up 2 sockets
Jun 20 00:15:03 dist-test-slave-28bv krb5kdc[13708](info): commencing operation
krb5kdc: starting...
W20260620 00:15:04.476465 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:15:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:15:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:15:04.604943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:04.690357 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:04.690649 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:04.690693 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:04.692096 13721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:04.692202 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:04.692232 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:04.692269 13721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:04.692303 13721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:04.694156 13721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:04.694480 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:04.694752 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:04.700311 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:04.700349 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:04.700482 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:04.701115 13721 server_base.cc:1034] running on GCE node
I20260620 00:15:04.701438 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:04.701833 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:04.703028 13721 hybrid_clock.cc:648] HybridClock initialized: now 1781914504702996 us; error 51 us; skew 500 ppm
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:04.709543 13721 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:04.710204 13721 webserver.cc:460] Webserver started at http://127.0.98.62:42477/ using document root <none> and password file <none>
I20260620 00:15:04.710474 13721 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:04.710553 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:04.710736 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:04.712025 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "a00b83abcecf4a91b3e00743d72fdabd"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "f18e9ddeab6e206e2d5fc975d694c7c7"
server_key_iv: "8cdb45de575410dac4a93b9cda322bf8"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.712467 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "a00b83abcecf4a91b3e00743d72fdabd"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "f18e9ddeab6e206e2d5fc975d694c7c7"
server_key_iv: "8cdb45de575410dac4a93b9cda322bf8"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.714028 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:04.714953 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:04.715305 13721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:04.715444 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "a00b83abcecf4a91b3e00743d72fdabd"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "f18e9ddeab6e206e2d5fc975d694c7c7"
server_key_iv: "8cdb45de575410dac4a93b9cda322bf8"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.715531 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:04.726603 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:04.729035 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:04.729228 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:04.733894 13721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34637
I20260620 00:15:04.734000 13788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34637 every 8 connection(s)
I20260620 00:15:04.734390 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:04.735447 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:04.737289 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:04.740407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13721
I20260620 00:15:04.740525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:15:04.740749 392 external_mini_cluster.cc:1428] Setting key dba4b7f481440a440775e35ffcbeeded
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 00:15:04.748903 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.738159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.98.62:34637)
0620 00:15:04.738394 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:04.738404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:04.738541 (+ 137us) server_negotiation.cc:407] Connection header received
0620 00:15:04.739192 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:04.739194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:04.739222 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:04.739329 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:04.739950 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.741389 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.742404 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.742753 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.742885 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.745299 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:04.745311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:04.745311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:04.745318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:04.747097 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.747553 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.747556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.747557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.747641 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.747881 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.747882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.747882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.748105 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:04.748264 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:04.748460 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 00:15:04.748644 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":173,"thread_start_us":106,"threads_started":1}
I20260620 00:15:04.748904 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.738128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.0.1:60976)
0620 00:15:04.738373 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.738424 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:04.739007 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:04.739335 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:04.739338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:04.739650 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:04.739827 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.739832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.741504 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.741505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.742275 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.742277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.742859 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.742860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.742937 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.743581 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:04.743590 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:04.745170 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:04.747247 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.747253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.747262 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.747452 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.747726 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.747727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.747727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.747813 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.748244 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:04.748248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:04.748322 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 00:15:04.748434 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:15:04.748715 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":56,"thread_start_us":102,"threads_started":1}
I20260620 00:15:04.749982 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:04.750471 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.750319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50076)
0620 00:15:04.750383 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.750435 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 00:15:04.750658 13739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
I20260620 00:15:04.751215 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.751113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50082)
0620 00:15:04.751158 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.751188 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:15:04.751348 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:15:04.771596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41269
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:04.800657 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } attempt: 1
I20260620 00:15:04.801193 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.801027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50098)
0620 00:15:04.801099 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.801154 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:15:04.801815 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.801602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50104)
0620 00:15:04.801721 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.801756 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 00:15:04.802062 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
W20260620 00:15:04.857762 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:04.857995 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:04.858021 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:04.859254 13796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:04.859308 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:04.859323 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:04.859336 13796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:04.859349 13796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:04.861032 13796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:04.861333 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:04.861600 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:04.866766 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:04.866786 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:04.866770 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:04.867082 13796 server_base.cc:1034] running on GCE node
I20260620 00:15:04.867358 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:04.867694 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:04.868873 13796 hybrid_clock.cc:648] HybridClock initialized: now 1781914504868867 us; error 37 us; skew 500 ppm
I20260620 00:15:04.870246 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } attempt: 2
I20260620 00:15:04.870622 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.870506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50108)
0620 00:15:04.870555 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.870591 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 00:15:04.871012 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.870909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50112)
0620 00:15:04.870962 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.870988 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:15:04.871124 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41269: connect: Connection refused (error 111)
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:04.871809 13796 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:15:04.872391 13796 webserver.cc:460] Webserver started at http://127.0.98.61:34729/ using document root <none> and password file <none>
I20260620 00:15:04.872748 13796 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:04.872805 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:04.872973 13796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:04.874138 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data/instance:
uuid: "0e1ca1fff6e44cdd954243a5b1d058f9"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "215bcb04d6762c2b0dfd6f8e9cd632f2"
server_key_iv: "f9d4d34eb7ddd7f2a8c8b62f3867f19a"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.874521 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal/instance:
uuid: "0e1ca1fff6e44cdd954243a5b1d058f9"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "215bcb04d6762c2b0dfd6f8e9cd632f2"
server_key_iv: "f9d4d34eb7ddd7f2a8c8b62f3867f19a"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.875988 13796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:15:04.876897 13811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:04.877090 13796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:04.877225 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "0e1ca1fff6e44cdd954243a5b1d058f9"
format_stamp: "Formatted at 2026-06-20 00:15:04 on dist-test-slave-28bv"
server_key: "215bcb04d6762c2b0dfd6f8e9cd632f2"
server_key_iv: "f9d4d34eb7ddd7f2a8c8b62f3867f19a"
server_key_version: "encryptionkey@0"
I20260620 00:15:04.877295 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:04.900663 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:04.903311 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:04.903507 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:04.907924 13796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41269
I20260620 00:15:04.908002 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41269 every 8 connection(s)
I20260620 00:15:04.908385 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:15:04.909348 13864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:04.910933 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:04.917359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13796
I20260620 00:15:04.917471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-1/wal/instance
I20260620 00:15:04.917687 392 external_mini_cluster.cc:1428] Setting key 0b71e12efc5c060127d745a4b6fc18d8
I20260620 00:15:04.921483 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.911719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.98.62:34637)
0620 00:15:04.911798 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:04.911801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:04.911979 (+ 178us) server_negotiation.cc:407] Connection header received
0620 00:15:04.912530 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:04.912532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:04.912578 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:04.912660 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:04.913370 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.913881 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.914909 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.915151 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.915227 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.917728 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:04.917730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:04.917730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:04.917739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:04.919407 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.920288 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.920290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.920291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.920386 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.920729 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.920730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.920730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.920864 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:04.920931 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:04.921313 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 00:15:04.921400 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:15:04.921767 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.911627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.0.1:60986)
0620 00:15:04.911853 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.911892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:04.912412 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:04.912745 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:04.912748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:04.913001 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:04.913258 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.913263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.913980 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.913981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.914805 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.914808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.915233 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.915234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.915364 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.915944 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:04.915953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:04.917578 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:04.919516 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.919525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.919528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.920166 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.920465 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.920466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.920467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.920663 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.920942 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:04.920949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:04.921141 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 00:15:04.921264 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:15:04.921468 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":107,"threads_started":1}
I20260620 00:15:04.922559 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:15:04.934340 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.922934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.0.98.61:41269)
0620 00:15:04.923109 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:04.923114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:04.923134 (+ 20us) server_negotiation.cc:407] Connection header received
0620 00:15:04.923162 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:04.923164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:04.923175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:04.923239 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:04.923641 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.924479 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.925586 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.925948 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.926018 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.930890 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:04.930907 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:04.930907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:04.930920 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:04.933007 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.933403 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.933407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.933407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.933464 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.933656 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.933657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.933657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.933818 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:04.933913 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:04.934049 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 00:15:04.934195 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260620 00:15:04.934343 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.922892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50120)
0620 00:15:04.922956 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.922970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:04.923022 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:04.923260 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:04.923262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:04.923451 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:04.923532 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.923534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.924628 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.924630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.925488 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.925490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.926041 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.926042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.926133 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.926648 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:04.926654 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:04.930760 (+ 4106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:04.933136 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.933137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.933137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.933303 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.933532 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.933533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.933534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.933592 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.933889 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:04.933890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:04.933929 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 00:15:04.934016 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 00:15:04.934246 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":128}
I20260620 00:15:04.935035 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:04.935529 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.935361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56864)
0620 00:15:04.935426 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.935490 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 00:15:04.935762 13813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
I20260620 00:15:04.936268 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.936152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56876)
0620 00:15:04.936207 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.936240 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:15:04.936461 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:15:04.948521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46761
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:04.962289 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } attempt: 3
Jun 20 00:15:04 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:15:04.969622 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.962673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50136)
0620 00:15:04.962730 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.962744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:04.962834 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:04.962993 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:04.962994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:04.963152 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:04.963222 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.963224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.963908 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.963910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.964432 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:04.964434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.964872 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.964873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:04.964948 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.965431 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:04.965436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:04.966831 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:04.968478 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.968479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.968480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.968742 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.968948 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:04.968949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:04.968949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:04.969007 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:04.969290 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:04.969291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:04.969337 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:15:04.969419 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:15:04.969519 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:15:04.969622 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.962760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.0.98.61:41269)
0620 00:15:04.962826 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:04.962828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:04.962839 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:15:04.962905 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:04.962905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:04.962917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:04.962987 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:04.963309 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.963819 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.964528 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:04.964766 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:04.964858 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:04.966921 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:04.966923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:04.966923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:04.966931 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:04.968375 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.968817 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.968818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.968819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.968879 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:04.969095 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:04.969096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:04.969096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:04.969209 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:04.969280 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:04.969444 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:15:04.969522 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:15:04.969985 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:04.970381 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.970281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56880)
0620 00:15:04.970334 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.970363 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:15:04.970650 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.970599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56890)
0620 00:15:04.970621 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.970639 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 00:15:04.970752 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
I20260620 00:15:04.985735 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 1
I20260620 00:15:04.986223 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.986083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56904)
0620 00:15:04.986151 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.986191 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:15:04.986596 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:04.986500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56908)
0620 00:15:04.986546 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:04.986570 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:15:04.986728 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
I20260620 00:15:05.001955 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 1
I20260620 00:15:05.002619 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.002470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56914)
0620 00:15:05.002544 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.002587 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 00:15:05.003010 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.002903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56920)
0620 00:15:05.002953 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.002988 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:15:05.003105 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
W20260620 00:15:05.034571 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:05.034847 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:05.034888 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:05.036063 13871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:05.036137 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:05.036162 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:05.036190 13871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:05.036221 13871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:05.037817 13871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34637,127.0.98.61:41269,127.0.98.60:46761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:05.038122 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:05.038364 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:05.043419 13876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:05.043466 13877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:05.043591 13879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:05.043488 13871 server_base.cc:1034] running on GCE node
I20260620 00:15:05.043919 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:05.044231 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:05.045390 13871 hybrid_clock.cc:648] HybridClock initialized: now 1781914505045332 us; error 78 us; skew 500 ppm
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:05.048362 13871 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:15:05.048962 13871 webserver.cc:460] Webserver started at http://127.0.98.60:44063/ using document root <none> and password file <none>
I20260620 00:15:05.049201 13871 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:05.049247 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:05.049383 13871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:05.050462 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data/instance:
uuid: "44275c02c8034371b6abaf6ca0a0be70"
format_stamp: "Formatted at 2026-06-20 00:15:05 on dist-test-slave-28bv"
server_key: "38f8f498e1b8189a24d6df4479a836bd"
server_key_iv: "7f71ffdfba284099d564f33451bdbd67"
server_key_version: "encryptionkey@0"
I20260620 00:15:05.050833 13871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal/instance:
uuid: "44275c02c8034371b6abaf6ca0a0be70"
format_stamp: "Formatted at 2026-06-20 00:15:05 on dist-test-slave-28bv"
server_key: "38f8f498e1b8189a24d6df4479a836bd"
server_key_iv: "7f71ffdfba284099d564f33451bdbd67"
server_key_version: "encryptionkey@0"
I20260620 00:15:05.052179 13871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:15:05.053057 13886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:05.053300 13871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:05.053412 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "44275c02c8034371b6abaf6ca0a0be70"
format_stamp: "Formatted at 2026-06-20 00:15:05 on dist-test-slave-28bv"
server_key: "38f8f498e1b8189a24d6df4479a836bd"
server_key_iv: "7f71ffdfba284099d564f33451bdbd67"
server_key_version: "encryptionkey@0"
I20260620 00:15:05.053496 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:05.054924 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 2
I20260620 00:15:05.055384 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.055237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56926)
0620 00:15:05.055305 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.055346 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:15:05.055729 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.055634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56940)
0620 00:15:05.055680 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.055707 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:15:05.055827 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
I20260620 00:15:05.078286 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 2
I20260620 00:15:05.078820 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.078675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56956)
0620 00:15:05.078743 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.078786 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:15:05.079239 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.079103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56968)
0620 00:15:05.079165 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.079207 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:15:05.079394 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46761: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46761: connect: Connection refused (error 111)
I20260620 00:15:05.090454 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:05.092391 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:05.092505 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:05.096659 13871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46761
I20260620 00:15:05.096745 13938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46761 every 8 connection(s)
I20260620 00:15:05.097108 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:15:05.097975 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:05.099606 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:05.105077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 13871
I20260620 00:15:05.105274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914322045185-392-0/minicluster-data/master-2/wal/instance
I20260620 00:15:05.105511 392 external_mini_cluster.cc:1428] Setting key 12d2deb2cb9232b00efcf56e53821c97
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:05.110895 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.100383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.0.98.62:34637)
0620 00:15:05.100486 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.100489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.100816 (+ 327us) server_negotiation.cc:407] Connection header received
0620 00:15:05.101313 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.101315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.101326 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.101396 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.102195 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.102994 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.104089 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.104291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.104359 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.107270 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.107272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.107273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.107281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.108966 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.109567 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.109568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.109569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.109694 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.110060 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.110061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.110062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.110188 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.110266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.110668 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.110794 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 00:15:05.111129 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.100350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34637 (local address 127.0.0.1:60994)
0620 00:15:05.100629 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.100703 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:05.101165 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:05.101509 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:05.101513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:05.101838 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:05.102063 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.102075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.103112 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.103113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.103989 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.103991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.104383 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.104384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.104503 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.105229 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:05.105239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:05.107180 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:05.109072 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.109080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.109082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.109463 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.109781 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.109782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.109783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.109960 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.110282 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:05.110290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:05.110491 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 00:15:05.110633 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 00:15:05.110842 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":124,"threads_started":1}
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:15:05.112134 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:15:05.116043 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.105991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.0.98.60:46761)
0620 00:15:05.106169 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.106174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.106226 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:15:05.106283 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.106285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.106303 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.106366 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.107093 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.108027 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.108840 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.109274 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.109335 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.112175 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.112183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.112184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.112191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.114281 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.114695 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.114698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.114698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.114768 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.115105 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.115106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.115107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.115350 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.115512 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.115625 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.115749 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:15:05.121141 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.112519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41269 (local address 127.0.0.1:50140)
0620 00:15:05.112596 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.112608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:05.112650 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:05.112810 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:05.112811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:05.113002 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:05.113071 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.113073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.113923 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.113924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.114671 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.114673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.115258 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.115259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.115353 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.115843 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:05.115848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:05.117372 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:05.119908 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.119909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.119910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.120139 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.120411 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.120413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.120413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.120498 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.120883 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:05.120884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:05.120928 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:15:05.120989 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:15:05.121071 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":7}
I20260620 00:15:05.121572 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.112642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.98.61:41269)
0620 00:15:05.112690 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.112692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.112700 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:15:05.112723 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.112724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.112734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.112806 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.113171 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.113837 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.114766 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.115167 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.115230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.117462 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.117463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.117464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.117473 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.119841 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.120210 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.120211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.120211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.120276 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.120603 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.120604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.120604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.120763 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.120858 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.121061 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.121170 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:15:05.121928 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:15:05.126036 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.117118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.98.62:34637)
0620 00:15:05.117176 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.117177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.117186 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:15:05.117211 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.117211 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.117221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.117281 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.117925 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.118665 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.119794 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.120057 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.120133 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.122983 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.122985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.122985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.122994 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.124826 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.125246 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.125248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.125248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.125313 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.125535 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.125535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.125536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.125639 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.125810 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.125845 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.125936 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:15:05.129176 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.122217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56992)
0620 00:15:05.122259 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.122269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:05.122315 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:05.122479 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:05.122480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:05.122634 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:05.122663 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.122665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.123365 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.123366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.123839 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.123841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.124213 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.124214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.124262 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.124745 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:05.124749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:05.126332 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:05.127966 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.127967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.127967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.128223 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.128455 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.128456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.128456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.128517 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.128867 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:05.128867 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:05.128903 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 00:15:05.128953 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:15:05.129066 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 00:15:05.129244 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.122317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.0.98.60:46761)
0620 00:15:05.122375 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.122377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.122383 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:15:05.122399 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.122399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.122409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.122479 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.122737 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.123283 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.123918 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.124130 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.124214 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.126432 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.126433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.126434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.126442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.127882 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.128296 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.128297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.128297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.128357 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.128661 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.128662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.128662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.128799 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.128865 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.129014 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.129128 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":39}
I20260620 00:15:05.131101 13939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: Bootstrap starting.
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:15:05.132048 13939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:05.132413 13939 log.cc:824] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:05.133384 13939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: No bootstrap required, opened a new log
I20260620 00:15:05.134840 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.126745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.0.98.61:41269)
0620 00:15:05.126810 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.126813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.126822 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:15:05.126889 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.126890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.126900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.126954 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.127490 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.128071 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.129056 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.129377 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.129628 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.132054 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.132056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.132056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.132064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.133670 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.134023 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.134024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.134025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.134082 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.134279 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.134280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.134280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.134425 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.134617 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.134656 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.134736 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:15:05.135003 13939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.135131 13939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:05.135201 13939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44275c02c8034371b6abaf6ca0a0be70, State: Initialized, Role: FOLLOWER
I20260620 00:15:05.135353 13939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.135700 13939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:05.135670 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.135757 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:15:05.136590 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 00:15:05.137744 13956 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:15:05.137799 13956 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:15:05.138019 13871 master_runner.cc:186] Error getting master registration for 127.0.98.62:34637: OK, instance_id { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" instance_seqno: 1781914504732474 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:15:05.141256 13871 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:15:05.146979 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 3
I20260620 00:15:05.150290 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.140906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32770 (local address 127.0.98.62:34637)
0620 00:15:05.140953 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.140955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.141084 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:15:05.141201 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.141202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.141216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.141249 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.142268 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.143015 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144114 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.144400 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144440 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.146544 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.146545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.146546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.146554 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.148940 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.149373 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.149375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.149375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.149436 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.149648 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.149648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.149649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.149797 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.149883 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.150103 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.150184 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:15:05.151029 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.141352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.98.61:41269)
0620 00:15:05.141390 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.141392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.141640 (+ 248us) server_negotiation.cc:407] Connection header received
0620 00:15:05.141738 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.141739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.141751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.141783 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.142540 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.143242 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144189 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.144471 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144507 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.146538 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.146539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.146539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.146547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.149708 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.150151 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.150153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.150153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.150216 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.150407 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.150408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.150408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.150535 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.150630 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.150860 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.150940 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:15:05.151448 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.141137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.0.98.60:46761)
0620 00:15:05.141175 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.141177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.141424 (+ 247us) server_negotiation.cc:407] Connection header received
0620 00:15:05.141517 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.141518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.141529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.141559 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.142264 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.143010 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144099 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.144424 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.144466 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.146555 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.146556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.146557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.146563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.148804 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.149232 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.149233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.149233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.149292 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.149496 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.149496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.149497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.149698 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.149801 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.151269 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.151352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:15:05.157531 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.147232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:56996)
0620 00:15:05.147299 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.147312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:05.147356 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:05.147672 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:05.147673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:05.148326 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:05.148377 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.148380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.150972 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.150975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.151750 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.151752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.152837 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.152838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.152898 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.153353 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:05.153358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:05.154587 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:05.156358 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.156359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.156360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.156607 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.156833 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.156834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.156835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.156894 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.157164 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:05.157165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:05.157220 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 00:15:05.157306 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:15:05.157427 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:15:05.157578 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.147342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56996 (local address 127.0.98.60:46761)
0620 00:15:05.147525 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.147527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.147535 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:15:05.147556 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.147557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.147569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.149983 (+ 2414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.150035 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.150868 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.152419 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.152745 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.152836 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.154687 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.154689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.154689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.154696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.156239 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.156700 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.156701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.156702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.156761 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.156968 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.156969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.156969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.157091 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.157171 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.157346 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.157428 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
I20260620 00:15:05.159732 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: Bootstrap starting.
I20260620 00:15:05.160487 13864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:05.160872 13864 log.cc:824] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:05.161825 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: No bootstrap required, opened a new log
I20260620 00:15:05.163825 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.163972 13864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:05.164006 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1ca1fff6e44cdd954243a5b1d058f9, State: Initialized, Role: FOLLOWER
I20260620 00:15:05.164155 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.164584 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:05.164498 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.164662 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:15:05.166515 13978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:15:05.166577 13978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:15:05.166958 13796 master_runner.cc:186] Error getting master registration for 127.0.98.62:34637: OK, instance_id { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" instance_seqno: 1781914504732474 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:15:05.169092 13796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:15:05.178645 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } attempt: 3
Jun 20 00:15:05 dist-test-slave-28bv krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:15:05.186691 13965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.179085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.0.98.60:46761)
0620 00:15:05.179129 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:05.179132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:05.179196 (+ 64us) server_negotiation.cc:407] Connection header received
0620 00:15:05.179227 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:05.179228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:05.179238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:05.179302 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:05.179626 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.180192 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.181023 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.181332 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.181408 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.183290 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:05.183292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:05.183292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:05.183300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:05.185326 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.185765 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.185766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.185767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.185836 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:05.186074 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:05.186075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:05.186075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:05.186218 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:05.186302 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:05.186484 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 00:15:05.186575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:15:05.186691 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:05.179005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46761 (local address 127.0.0.1:57000)
0620 00:15:05.179063 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:05.179076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:05.179124 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:05.179320 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:05.179321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:05.179489 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:05.179532 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.179533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.180274 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.180275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.180932 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:05.180935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:05.181418 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:05.181420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:05.181485 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:05.181956 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:05.181964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:05.183193 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:05.185418 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.185420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.185420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.185670 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.185916 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:05.185917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:05.185917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:05.185994 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:05.186305 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:05.186306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:05.186360 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 00:15:05.186456 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 00:15:05.186573 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:15:05.188508 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: Bootstrap starting.
I20260620 00:15:05.189265 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:05.189585 13789 log.cc:824] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:05.190418 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: No bootstrap required, opened a new log
I20260620 00:15:05.191944 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.192102 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:05.192179 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00b83abcecf4a91b3e00743d72fdabd, State: Initialized, Role: FOLLOWER
I20260620 00:15:05.192363 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.192713 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.192775 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:05.192865 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:15:05.195847 13992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:15:05.195971 13992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:15:05.198267 13721 master_runner.cc:263] No leader master found from master a00b83abcecf4a91b3e00743d72fdabd
I20260620 00:15:05.198330 13721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:15:05.219297 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:05.219455 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.219755 13967 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a00b83abcecf4a91b3e00743d72fdabd (127.0.98.62:34637), 44275c02c8034371b6abaf6ca0a0be70 (127.0.98.60:46761)
I20260620 00:15:05.220230 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44275c02c8034371b6abaf6ca0a0be70" is_pre_election: true
I20260620 00:15:05.220418 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1ca1fff6e44cdd954243a5b1d058f9 in term 0.
I20260620 00:15:05.220630 13813 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1ca1fff6e44cdd954243a5b1d058f9, 44275c02c8034371b6abaf6ca0a0be70; no voters:
I20260620 00:15:05.221066 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:05.221138 13967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:05.221160 13967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:05.221159 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00b83abcecf4a91b3e00743d72fdabd" is_pre_election: true
I20260620 00:15:05.221342 13763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e1ca1fff6e44cdd954243a5b1d058f9 in term 0.
I20260620 00:15:05.222218 13967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.222362 13967 leader_election.cc:288] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [CANDIDATE]: Term 1 election: Requested vote from peers a00b83abcecf4a91b3e00743d72fdabd (127.0.98.62:34637), 44275c02c8034371b6abaf6ca0a0be70 (127.0.98.60:46761)
I20260620 00:15:05.222541 13763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a00b83abcecf4a91b3e00743d72fdabd"
I20260620 00:15:05.222626 13763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:05.222782 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44275c02c8034371b6abaf6ca0a0be70"
I20260620 00:15:05.222870 13913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:05.223584 13763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1ca1fff6e44cdd954243a5b1d058f9 in term 1.
I20260620 00:15:05.223687 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e1ca1fff6e44cdd954243a5b1d058f9 in term 1.
I20260620 00:15:05.223753 13813 leader_election.cc:302] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1ca1fff6e44cdd954243a5b1d058f9, a00b83abcecf4a91b3e00743d72fdabd; no voters:
I20260620 00:15:05.223948 13967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:05.224105 13967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e1ca1fff6e44cdd954243a5b1d058f9, State: Running, Role: LEADER
I20260620 00:15:05.224202 13967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } }
I20260620 00:15:05.227372 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1ca1fff6e44cdd954243a5b1d058f9. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.227560 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:05.227840 13996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:05.228065 13996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 00:15:05.230118 13742 tablet.cc:2369] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:05.230954 13913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [term 1 FOLLOWER]: Refusing update from remote peer 0e1ca1fff6e44cdd954243a5b1d058f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:05.231369 13994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:05.232910 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1ca1fff6e44cdd954243a5b1d058f9. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.233000 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:15:05.233491 13763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [term 1 FOLLOWER]: Refusing update from remote peer 0e1ca1fff6e44cdd954243a5b1d058f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:05.233795 13994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:05.234983 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e1ca1fff6e44cdd954243a5b1d058f9. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.235085 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:15:05.236618 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.236716 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:05.236698 13997 mvcc.cc:205] Tried to move back new op lower bound from 7298721813423833088 to 7298721813399003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298721813423833088}]
I20260620 00:15:05.237317 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.237394 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:15:05.237686 13998 mvcc.cc:205] Tried to move back new op lower bound from 7298721813423833088 to 7298721813399003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298721813423833088}]
I20260620 00:15:05.238940 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.239032 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:05.239504 13981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a00b83abcecf4a91b3e00743d72fdabd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34637 } } peers { permanent_uuid: "0e1ca1fff6e44cdd954243a5b1d058f9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41269 } } peers { permanent_uuid: "44275c02c8034371b6abaf6ca0a0be70" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46761 } } }
I20260620 00:15:05.239615 13981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:15:05.240741 14000 mvcc.cc:205] Tried to move back new op lower bound from 7298721813423833088 to 7298721813399003136. Current Snapshot: MvccSnapshot[applied={T|T < 7298721813423833088}]
I20260620 00:15:05.241328 13996 catalog_manager.cc:1347] Generated new cluster ID: ba9e849c15b14ae48a32631bc8687918
I20260620 00:15:05.241375 13996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:05.258376 13996 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:15:05.259202 13996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:05.267030 13996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e1ca1fff6e44cdd954243a5b1d058f9: Generated new TSK 0
I20260620 00:15:05.267354 13996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:15:06.138334 13956 catalog_manager.cc:1259] Loaded cluster ID: ba9e849c15b14ae48a32631bc8687918
I20260620 00:15:06.138432 13956 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: loading cluster ID for follower catalog manager: success
I20260620 00:15:06.139313 13956 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: acquiring CA information for follower catalog manager: success
I20260620 00:15:06.139627 13956 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44275c02c8034371b6abaf6ca0a0be70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:15:06.196519 13992 catalog_manager.cc:1259] Loaded cluster ID: ba9e849c15b14ae48a32631bc8687918
I20260620 00:15:06.196699 13992 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: loading cluster ID for follower catalog manager: success
I20260620 00:15:06.197585 13992 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: acquiring CA information for follower catalog manager: success
I20260620 00:15:06.197906 13992 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a00b83abcecf4a91b3e00743d72fdabd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:15:06.320124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13721
I20260620 00:15:06.326088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13796
I20260620 00:15:06.331331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 13871
2026-06-20T00:15:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2908 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:15:06 dist-test-slave-28bv krb5kdc[14013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:15:06 dist-test-slave-28bv krb5kdc[14013](info): set up 2 sockets
Jun 20 00:15:06 dist-test-slave-28bv krb5kdc[14013](info): commencing operation
krb5kdc: starting...
W20260620 00:15:07.388924 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:15:07.515791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40279
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:07.598517 14026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:07.598747 14026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:07.598770 14026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:07.599880 14026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:07.599918 14026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:07.599932 14026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:07.599946 14026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:07.599958 14026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:07.601567 14026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:07.601786 14026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:07.601969 14026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:07.607306 14031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:07.607357 14032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:07.607359 14034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:07.607750 14026 server_base.cc:1034] running on GCE node
I20260620 00:15:07.608031 14026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:07.608352 14026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:07.609560 14026 hybrid_clock.cc:648] HybridClock initialized: now 1781914507609536 us; error 40 us; skew 500 ppm
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:07.612516 14026 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:07.613076 14026 webserver.cc:460] Webserver started at http://127.0.98.62:41549/ using document root <none> and password file <none>
I20260620 00:15:07.613303 14026 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:07.613351 14026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:07.613451 14026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:07.614578 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "494e7dd3e2d347248f62a92a81611725"
format_stamp: "Formatted at 2026-06-20 00:15:07 on dist-test-slave-28bv"
server_key: "2d3cb0ad107b68d47158518ab88bc138"
server_key_iv: "3c60cb022fa9f681cce686676c7a54fa"
server_key_version: "encryptionkey@0"
I20260620 00:15:07.614929 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "494e7dd3e2d347248f62a92a81611725"
format_stamp: "Formatted at 2026-06-20 00:15:07 on dist-test-slave-28bv"
server_key: "2d3cb0ad107b68d47158518ab88bc138"
server_key_iv: "3c60cb022fa9f681cce686676c7a54fa"
server_key_version: "encryptionkey@0"
I20260620 00:15:07.616802 14026 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:15:07.624012 14041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:07.624264 14026 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 00:15:07.624382 14026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "494e7dd3e2d347248f62a92a81611725"
format_stamp: "Formatted at 2026-06-20 00:15:07 on dist-test-slave-28bv"
server_key: "2d3cb0ad107b68d47158518ab88bc138"
server_key_iv: "3c60cb022fa9f681cce686676c7a54fa"
server_key_version: "encryptionkey@0"
I20260620 00:15:07.624495 14026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:07.639650 14026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:07.642287 14026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:07.642410 14026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:07.646713 14026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40279
I20260620 00:15:07.646752 14093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40279 every 8 connection(s)
I20260620 00:15:07.647114 14026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:07.648020 14094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:07.650359 14094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725: Bootstrap starting.
I20260620 00:15:07.650960 14094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:07.651046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14026
I20260620 00:15:07.651170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:15:07.651340 14094 log.cc:824] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:07.651374 392 external_mini_cluster.cc:1428] Setting key 07169a873a5142fe5b727ba092a1eb12
I20260620 00:15:07.652201 14094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725: No bootstrap required, opened a new log
I20260620 00:15:07.654848 14094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260620 00:15:07.655035 14094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:07.655071 14094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494e7dd3e2d347248f62a92a81611725, State: Initialized, Role: FOLLOWER
I20260620 00:15:07.655200 14094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260620 00:15:07.655277 14094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:07.655313 14094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:07.655377 14094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:07.656081 14094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260620 00:15:07.656172 14094 leader_election.cc:302] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 494e7dd3e2d347248f62a92a81611725; no voters:
I20260620 00:15:07.656396 14094 leader_election.cc:288] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:15:07.656590 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:07.656692 14094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:07.657163 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [term 1 LEADER]: Becoming Leader. State: Replica: 494e7dd3e2d347248f62a92a81611725, State: Running, Role: LEADER
I20260620 00:15:07.657269 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260620 00:15:07.657670 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "494e7dd3e2d347248f62a92a81611725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } } }
I20260620 00:15:07.657763 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:07.657693 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494e7dd3e2d347248f62a92a81611725. Latest consensus state: current_term: 1 leader_uuid: "494e7dd3e2d347248f62a92a81611725" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494e7dd3e2d347248f62a92a81611725" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } } }
I20260620 00:15:07.657811 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725 [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:07.657981 14109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:07.658499 14109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:07.660934 14109 catalog_manager.cc:1347] Generated new cluster ID: 53f4d280fa034a7ba7875974fae70d15
I20260620 00:15:07.660993 14109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:07.663772 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:07.651916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.62:40279)
0620 00:15:07.652098 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:07.652106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:07.652139 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:15:07.652707 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:07.652714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:07.652773 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:07.652893 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:07.653745 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:07.654662 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:07.655749 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:07.656185 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:07.656305 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:07.659779 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:07.659783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:07.659784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:07.659797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:07.661715 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:07.662221 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:07.662225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:07.662226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:07.662383 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:07.662561 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:07.662561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:07.662562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:07.662859 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:07.663006 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:07.663433 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 00:15:07.663570 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 00:15:07.664808 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:15:07.677119 14109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:15:07.677937 14109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:07.691846 14109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494e7dd3e2d347248f62a92a81611725: Generated new TSK 0
I20260620 00:15:07.692090 14109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:15:07.718829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14026
2026-06-20T00:15:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1385 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14121](info): set up 2 sockets
Jun 20 00:15:07 dist-test-slave-28bv krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260620 00:15:08.776796 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:15:08 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:15:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:15:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:15:08.905503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:08.993186 14134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:08.993459 14134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:08.993486 14134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:08.994741 14134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:08.994789 14134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:08.994803 14134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:08.994817 14134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:08.994828 14134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:08.996455 14134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:08.996783 14134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:08.997026 14134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:09.002796 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.002835 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.002943 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.003036 14134 server_base.cc:1034] running on GCE node
I20260620 00:15:09.003342 14134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:09.003629 14134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:09.004803 14134 hybrid_clock.cc:648] HybridClock initialized: now 1781914509004783 us; error 33 us; skew 500 ppm
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:09.007820 14134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:09.008332 14134 webserver.cc:460] Webserver started at http://127.0.98.62:36367/ using document root <none> and password file <none>
I20260620 00:15:09.008646 14134 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:09.008702 14134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:09.008877 14134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:09.010090 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "f9c21907f3974680bc1c88153873c19e"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "554b71941b40aa0e16cfba7c6a26025f"
server_key_iv: "12c95641b493eaf3650cf3bd3295b8be"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.010524 14134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9c21907f3974680bc1c88153873c19e"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "554b71941b40aa0e16cfba7c6a26025f"
server_key_iv: "12c95641b493eaf3650cf3bd3295b8be"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.012347 14134 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:15:09.013798 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.014191 14134 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:09.014302 14134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "f9c21907f3974680bc1c88153873c19e"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "554b71941b40aa0e16cfba7c6a26025f"
server_key_iv: "12c95641b493eaf3650cf3bd3295b8be"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.014427 14134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:09.042734 14134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:09.045513 14134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:09.045647 14134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:09.049952 14134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260620 00:15:09.049983 14201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260620 00:15:09.050386 14134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:09.051440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14134
I20260620 00:15:09.051440 14202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:09.051582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:15:09.051795 392 external_mini_cluster.cc:1428] Setting key 7f615bbe316a80243ce59056400c2875
I20260620 00:15:09.055222 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap starting.
I20260620 00:15:09.055987 14202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:09.056313 14202 log.cc:824] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:09.057224 14202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: No bootstrap required, opened a new log
I20260620 00:15:09.059368 14202 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.059569 14202 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:09.059631 14202 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Initialized, Role: FOLLOWER
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:09.059806 14202 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.059921 14202 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:09.059958 14202 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:09.060010 14202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:09.061024 14202 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.061162 14202 leader_election.cc:302] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c21907f3974680bc1c88153873c19e; no voters:
I20260620 00:15:09.061388 14202 leader_election.cc:288] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:15:09.061424 14207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:09.061543 14207 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 LEADER]: Becoming Leader. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Running, Role: LEADER
I20260620 00:15:09.061718 14202 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:09.061637 14207 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.062474 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:09.062578 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:09.062469 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c21907f3974680bc1c88153873c19e. Latest consensus state: current_term: 1 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:09.062860 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:09.063392 14216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:09.063913 14216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:15:09.065227 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.052440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.0.98.62:43019)
0620 00:15:09.052665 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.052675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.052719 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:15:09.053296 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.053306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.053378 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.053551 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:09.054287 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.055716 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.056641 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.057221 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.057358 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.060673 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:09.060678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:09.060678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:09.060692 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:09.063104 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.063850 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.063853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.063854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.063990 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.064237 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.064238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.064238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.064524 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:09.064728 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.064839 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.064986 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":122,"threads_started":1}
I20260620 00:15:09.069180 14216 catalog_manager.cc:1347] Generated new cluster ID: 769f2a177a574dbcb158337ab57d7b19
I20260620 00:15:09.069309 14216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:09.081804 14216 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:15:09.082389 14216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:09.092154 14216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Generated new TSK 0
I20260620 00:15:09.092440 14216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:15:09.127905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:09.223392 14228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:09.223694 14228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:09.223747 14228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:09.225430 14228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:09.225524 14228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:09.225575 14228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:09.227349 14228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:09.227669 14228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:09.227928 14228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:09.233760 14233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.233760 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.234069 14234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.234079 14228 server_base.cc:1034] running on GCE node
I20260620 00:15:09.234360 14228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:09.234683 14228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:09.235870 14228 hybrid_clock.cc:648] HybridClock initialized: now 1781914509235847 us; error 51 us; skew 500 ppm
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:09.239068 14228 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:09.239673 14228 webserver.cc:460] Webserver started at http://127.0.98.1:37951/ using document root <none> and password file <none>
I20260620 00:15:09.239934 14228 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:09.239987 14228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:09.240129 14228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:09.241412 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b863d328275a145a5740cbb3e93c90d4"
server_key_iv: "064eda5384dabee6bddfc737db293cc0"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.241822 14228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b863d328275a145a5740cbb3e93c90d4"
server_key_iv: "064eda5384dabee6bddfc737db293cc0"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.243444 14228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:15:09.244663 14243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.245059 14228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:09.245303 14228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b863d328275a145a5740cbb3e93c90d4"
server_key_iv: "064eda5384dabee6bddfc737db293cc0"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.245446 14228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:09.270226 14228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:09.272948 14228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:09.273154 14228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:09.273428 14228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:09.273762 14228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:09.273820 14228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.273865 14228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:09.273891 14228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.280031 14228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37443
I20260620 00:15:09.280083 14356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37443 every 8 connection(s)
I20260620 00:15:09.280469 14228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:15:09.284557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14228
I20260620 00:15:09.284749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:15:09.285003 392 external_mini_cluster.cc:1428] Setting key 9249f9020d703e707d6ae199c316bafe
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:09.292027 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.281311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37059 (local address 127.0.98.62:43019)
0620 00:15:09.281445 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.281448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.281604 (+ 156us) server_negotiation.cc:407] Connection header received
0620 00:15:09.282162 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.282163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.282176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.282230 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:09.283257 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.283773 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.284970 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.285263 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.285340 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.288377 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:09.288379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:09.288380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:09.288390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:09.289960 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.290733 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.290734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.290734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.290801 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.291197 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.291198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.291198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.291390 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:09.291457 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.291873 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.291955 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:15:09.292358 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.281264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:37059)
0620 00:15:09.281454 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.281503 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.282052 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.282316 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.282320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.282936 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:09.283144 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.283149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.283889 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.283889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.284894 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.284897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.285421 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.285423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.285574 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.286205 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:09.286217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:09.288249 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:09.290201 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.290215 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.290219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.290614 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.290910 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.290911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.290911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.291094 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.291476 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:09.291484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:09.291698 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.291841 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.292057 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 00:15:09.292999 14357 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:09.293116 14357 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:09.293334 14357 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:09.293917 14166 ts_manager.cc:194] Registered new tserver with Master: 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443)
I20260620 00:15:09.294548 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37059
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:15:09.317173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:09.402652 14362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:09.402890 14362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:09.402917 14362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:09.404264 14362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:09.404325 14362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:09.404356 14362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:09.406009 14362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:09.406328 14362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:09.406589 14362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:09.412640 14370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.412638 14368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.412700 14362 server_base.cc:1034] running on GCE node
W20260620 00:15:09.412634 14367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.413132 14362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:09.413430 14362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:09.414583 14362 hybrid_clock.cc:648] HybridClock initialized: now 1781914509414558 us; error 54 us; skew 500 ppm
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:09.417552 14362 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:09.418154 14362 webserver.cc:460] Webserver started at http://127.0.98.2:39727/ using document root <none> and password file <none>
I20260620 00:15:09.418407 14362 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:09.418457 14362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:09.418632 14362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:09.419777 14362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "564fa8d4587943c48cb2fe412639227f"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b666aff3d55c3f66676fcf929474c567"
server_key_iv: "33688b393db46c589499b7717aa2cbe2"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.420158 14362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "564fa8d4587943c48cb2fe412639227f"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b666aff3d55c3f66676fcf929474c567"
server_key_iv: "33688b393db46c589499b7717aa2cbe2"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.421681 14362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:15:09.422567 14377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.422748 14362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:09.422861 14362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "564fa8d4587943c48cb2fe412639227f"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b666aff3d55c3f66676fcf929474c567"
server_key_iv: "33688b393db46c589499b7717aa2cbe2"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.422946 14362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:09.435256 14362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:09.437920 14362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:09.438105 14362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:09.438416 14362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:09.438764 14362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:09.438822 14362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.438867 14362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:09.438892 14362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.445500 14362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38573
I20260620 00:15:09.445631 14490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38573 every 8 connection(s)
I20260620 00:15:09.445953 14362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:09.453122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14362
I20260620 00:15:09.453254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:15:09.453476 392 external_mini_cluster.cc:1428] Setting key 9c4c85d9ff76154c4d45e5b8be5eef4d
I20260620 00:15:09.458158 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.446833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45501 (local address 127.0.98.62:43019)
0620 00:15:09.446886 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.446888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.447243 (+ 355us) server_negotiation.cc:407] Connection header received
0620 00:15:09.447919 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.447921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.447934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.447978 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:09.449171 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.449687 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.450851 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.451055 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.451141 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.453864 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:09.453866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:09.453867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:09.453877 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:09.455816 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.456639 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.456640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.456640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.456711 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.457123 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.457124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.457125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.457379 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:09.457470 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.457880 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.457999 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:15:09.458197 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.446749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:45501)
0620 00:15:09.447093 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.447136 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.447774 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.448082 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.448086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.448796 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:09.449043 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.449049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.449799 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.449800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.450738 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.450741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.451149 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.451151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.451292 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.451913 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:09.451923 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:09.453712 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:09.455973 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.455986 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.455990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.456446 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.456830 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.456831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.456831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.457016 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.457543 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:09.457551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:09.457713 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.457824 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.457993 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":237,"threads_started":1}
I20260620 00:15:09.459088 14491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:09.459496 14491 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:09.459829 14491 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:09.460717 14166 ts_manager.cc:194] Registered new tserver with Master: 564fa8d4587943c48cb2fe412639227f (127.0.98.2:38573)
I20260620 00:15:09.461237 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45501
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:15:09.488708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:09.577203 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:09.577473 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:09.577518 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:09.578838 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:09.578924 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:09.578975 14496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:09.580760 14496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:09.581063 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:09.581322 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:09.586982 14501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.586978 14502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.586951 14504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.587263 14496 server_base.cc:1034] running on GCE node
I20260620 00:15:09.587522 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:09.587756 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:09.588923 14496 hybrid_clock.cc:648] HybridClock initialized: now 1781914509588913 us; error 42 us; skew 500 ppm
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:09.591902 14496 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:09.592517 14496 webserver.cc:460] Webserver started at http://127.0.98.3:45119/ using document root <none> and password file <none>
I20260620 00:15:09.592873 14496 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:09.592947 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:09.593048 14496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:09.594189 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "457fab3163fc45938521f784ca461eff"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "7d568996395747ace8ee2263faec58d6"
server_key_iv: "6c436262dee14f84ee25e05cac1c3f25"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.594578 14496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "457fab3163fc45938521f784ca461eff"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "7d568996395747ace8ee2263faec58d6"
server_key_iv: "6c436262dee14f84ee25e05cac1c3f25"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.595948 14496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:15:09.596865 14511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.597091 14496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:09.597186 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "457fab3163fc45938521f784ca461eff"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "7d568996395747ace8ee2263faec58d6"
server_key_iv: "6c436262dee14f84ee25e05cac1c3f25"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.597250 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:09.609771 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:09.612160 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:09.612301 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:09.612525 14496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:09.612955 14496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:09.612991 14496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.613015 14496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:09.613061 14496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.619825 14496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42711
I20260620 00:15:09.619928 14624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42711 every 8 connection(s)
I20260620 00:15:09.620271 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:15:09.624598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14496
I20260620 00:15:09.624706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:15:09.624910 392 external_mini_cluster.cc:1428] Setting key 577ca3bc137d6d86c2c40849d0c672fc
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:09.631898 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.621218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58897 (local address 127.0.98.62:43019)
0620 00:15:09.621307 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.621310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.621513 (+ 203us) server_negotiation.cc:407] Connection header received
0620 00:15:09.622115 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.622117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.622129 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.622195 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:09.623322 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.623890 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.625308 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.625530 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.625654 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.628136 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:09.628138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:09.628138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:09.628154 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:09.629899 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.630678 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.630680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.630681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.630765 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.631076 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.631077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.631078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.631217 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:09.631370 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.631699 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.631799 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:15:09.632148 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.621088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:58897)
0620 00:15:09.621354 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.621413 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.621989 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.622282 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.622287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.622970 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:09.623193 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.623199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.624007 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.624008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.625198 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.625207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.625663 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.625664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.625824 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.626467 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:09.626477 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:09.628002 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:09.630230 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.630243 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.630246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.630572 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.630866 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:09.630867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:09.630867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:09.630994 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:09.631369 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:09.631377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:09.631550 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.631664 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.631859 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
I20260620 00:15:09.632828 14625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:09.632957 14625 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:09.633175 14625 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:09.633669 14166 ts_manager.cc:194] Registered new tserver with Master: 457fab3163fc45938521f784ca461eff (127.0.98.3:42711)
I20260620 00:15:09.634093 14166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58897
I20260620 00:15:09.636682 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:15:09.644769 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.638119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.98.62:43019)
0620 00:15:09.638188 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.638191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.638323 (+ 132us) server_negotiation.cc:407] Connection header received
0620 00:15:09.638368 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.638369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.638379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.638420 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:09.639025 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.639560 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.640364 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.640636 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.640697 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.642053 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:09.642055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:09.642055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:09.642062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:09.643442 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.643874 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.643875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.643876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.643940 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:09.644131 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:09.644131 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:09.644132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:09.644297 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:09.644438 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.644532 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.644692 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:15:09.647033 14166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:15:09.648715 14166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:15:09.656231 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.651703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38573 (local address 127.0.0.1:49994)
0620 00:15:09.652069 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.652080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.652158 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.652599 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.652600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.652603 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:09.652680 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.652682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.654006 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.654007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.655126 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.655128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.655834 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.655965 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.655978 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.656080 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.656142 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":12,"thread_start_us":109,"threads_started":1}
I20260620 00:15:09.656250 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.651896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.0.98.2:38573)
0620 00:15:09.652046 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.652051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.652132 (+ 81us) server_negotiation.cc:407] Connection header received
0620 00:15:09.652379 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.652382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.652399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.652469 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:09.652768 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.653885 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655223 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.655735 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.655896 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.656099 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.656142 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260620 00:15:09.656247 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.651733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.0.98.3:42711)
0620 00:15:09.651934 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.651940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.652148 (+ 208us) server_negotiation.cc:407] Connection header received
0620 00:15:09.652312 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.652315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.652335 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.652419 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:09.652751 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.653479 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.654962 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.655744 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655819 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.655885 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.656099 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.656147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260620 00:15:09.656394 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.651888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37443 (local address 127.0.0.1:43056)
0620 00:15:09.652351 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.652361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.652408 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.652846 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.652847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.652849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:09.652887 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.652888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.654109 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.654110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.655157 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.655160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.656147 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.656148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.656217 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.656221 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.656271 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.656320 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":86,"threads_started":1}
I20260620 00:15:09.656490 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.651632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.0.1:60620)
0620 00:15:09.652031 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.652075 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.652202 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.652579 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.652582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.652584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:09.652668 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.652670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.653587 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.653589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.654836 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.654839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.655924 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.655994 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.655999 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.656346 (+ 347us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.656388 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":75,"threads_started":1}
I20260620 00:15:09.656679 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.652129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.0.98.1:37443)
0620 00:15:09.652611 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.652616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.652625 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:15:09.652664 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.652667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.652681 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.652767 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:09.652966 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.654014 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.655279 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.656003 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.656062 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.656116 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.656501 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.656584 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":140,"threads_started":1}
I20260620 00:15:09.657481 14291 tablet_service.cc:1467] Processing CreateTablet for tablet 59178bf341c54ea991631b85e8707c3d (DEFAULT_TABLE table=test-table [id=b977c4e9b3a4418ea467789918a848a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:09.657480 14559 tablet_service.cc:1467] Processing CreateTablet for tablet 59178bf341c54ea991631b85e8707c3d (DEFAULT_TABLE table=test-table [id=b977c4e9b3a4418ea467789918a848a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:09.657879 14291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59178bf341c54ea991631b85e8707c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:09.657879 14559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59178bf341c54ea991631b85e8707c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:09.657902 14425 tablet_service.cc:1467] Processing CreateTablet for tablet 59178bf341c54ea991631b85e8707c3d (DEFAULT_TABLE table=test-table [id=b977c4e9b3a4418ea467789918a848a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:09.658249 14425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59178bf341c54ea991631b85e8707c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:09.661159 14642 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Bootstrap starting.
I20260620 00:15:09.661537 14644 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Bootstrap starting.
I20260620 00:15:09.662015 14642 tablet_bootstrap.cc:654] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:09.662125 14644 tablet_bootstrap.cc:654] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:09.662523 14642 log.cc:824] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:09.662633 14644 log.cc:824] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:09.662997 14643 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Bootstrap starting.
I20260620 00:15:09.663393 14642 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: No bootstrap required, opened a new log
I20260620 00:15:09.663478 14642 ts_tablet_manager.cc:1397] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:15:09.663730 14643 tablet_bootstrap.cc:654] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:09.664248 14643 log.cc:824] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:09.664855 14644 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: No bootstrap required, opened a new log
I20260620 00:15:09.664942 14644 ts_tablet_manager.cc:1397] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:15:09.665071 14643 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: No bootstrap required, opened a new log
I20260620 00:15:09.665155 14643 ts_tablet_manager.cc:1397] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:15:09.665433 14642 raft_consensus.cc:348] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.665637 14642 raft_consensus.cc:374] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:09.665670 14642 raft_consensus.cc:729] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564fa8d4587943c48cb2fe412639227f, State: Initialized, Role: FOLLOWER
I20260620 00:15:09.665817 14642 consensus_queue.cc:260] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.666136 14642 ts_tablet_manager.cc:1428] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:15:09.666157 14491 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 00:15:09.666894 14644 raft_consensus.cc:348] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.667080 14644 raft_consensus.cc:374] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:09.667127 14644 raft_consensus.cc:729] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0624e3bcc7a94ad39f852a8fd35ef7cd, State: Initialized, Role: FOLLOWER
I20260620 00:15:09.667264 14644 consensus_queue.cc:260] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.667218 14643 raft_consensus.cc:348] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.667407 14643 raft_consensus.cc:374] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:09.667446 14643 raft_consensus.cc:729] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457fab3163fc45938521f784ca461eff, State: Initialized, Role: FOLLOWER
I20260620 00:15:09.667588 14644 ts_tablet_manager.cc:1428] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:15:09.667646 14643 consensus_queue.cc:260] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.667714 14357 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 00:15:09.667954 14625 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 00:15:09.667937 14643 ts_tablet_manager.cc:1428] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 00:15:09.696722 14492 tablet.cc:2369] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:09.722144 14649 raft_consensus.cc:482] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:09.722288 14649 raft_consensus.cc:504] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.722671 14649 leader_election.cc:288] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457fab3163fc45938521f784ca461eff (127.0.98.3:42711), 564fa8d4587943c48cb2fe412639227f (127.0.98.2:38573)
I20260620 00:15:09.725925 14359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.722882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.98.1:46513)
0620 00:15:09.722930 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.722947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.723004 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.723223 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.723224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.723226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:09.723306 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.723308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.724049 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.724050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.724885 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.724887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.725611 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.725612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.725697 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.725701 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.725763 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.725815 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:15:09.725942 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.723008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46513 (local address 127.0.98.3:42711)
0620 00:15:09.723072 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.723074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.723085 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:15:09.723120 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.723121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.723131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.723203 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:09.723411 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.723962 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.724970 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.725524 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.725601 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.725613 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.725794 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.725837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:15:09.726193 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.722996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41379 (local address 127.0.98.2:38573)
0620 00:15:09.723071 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.723074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.723279 (+ 205us) server_negotiation.cc:407] Connection header received
0620 00:15:09.723326 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.723327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.723342 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.723397 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:09.723571 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.724078 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.725215 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.725754 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.725829 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.725839 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.726074 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.726125 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:15:09.726457 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.722892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38573 (local address 127.0.98.1:41379)
0620 00:15:09.723215 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.723232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:09.723279 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:09.723427 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:09.723428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:09.723430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:09.723496 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.723498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.724182 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.724183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.725123 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:09.725126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.725885 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.725886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:09.725968 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.725972 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:09.726269 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 00:15:09.726320 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":138,"thread_start_us":135,"threads_started":1}
I20260620 00:15:09.726377 14579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59178bf341c54ea991631b85e8707c3d" candidate_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457fab3163fc45938521f784ca461eff" is_pre_election: true
I20260620 00:15:09.726581 14579 raft_consensus.cc:2454] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0624e3bcc7a94ad39f852a8fd35ef7cd in term 0.
I20260620 00:15:09.726781 14445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59178bf341c54ea991631b85e8707c3d" candidate_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564fa8d4587943c48cb2fe412639227f" is_pre_election: true
I20260620 00:15:09.726845 14244 leader_election.cc:302] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0624e3bcc7a94ad39f852a8fd35ef7cd, 457fab3163fc45938521f784ca461eff; no voters:
I20260620 00:15:09.726950 14445 raft_consensus.cc:2454] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0624e3bcc7a94ad39f852a8fd35ef7cd in term 0.
I20260620 00:15:09.727011 14649 raft_consensus.cc:2781] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:09.727087 14649 raft_consensus.cc:482] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:09.727120 14649 raft_consensus.cc:3037] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:09.727847 14649 raft_consensus.cc:504] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.728015 14649 leader_election.cc:288] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [CANDIDATE]: Term 1 election: Requested vote from peers 457fab3163fc45938521f784ca461eff (127.0.98.3:42711), 564fa8d4587943c48cb2fe412639227f (127.0.98.2:38573)
I20260620 00:15:09.728211 14445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59178bf341c54ea991631b85e8707c3d" candidate_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564fa8d4587943c48cb2fe412639227f"
I20260620 00:15:09.728296 14445 raft_consensus.cc:3037] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:09.728294 14579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59178bf341c54ea991631b85e8707c3d" candidate_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457fab3163fc45938521f784ca461eff"
I20260620 00:15:09.728416 14579 raft_consensus.cc:3037] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:09.729099 14445 raft_consensus.cc:2454] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0624e3bcc7a94ad39f852a8fd35ef7cd in term 1.
I20260620 00:15:09.729274 14579 raft_consensus.cc:2454] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0624e3bcc7a94ad39f852a8fd35ef7cd in term 1.
I20260620 00:15:09.729393 14247 leader_election.cc:302] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0624e3bcc7a94ad39f852a8fd35ef7cd, 564fa8d4587943c48cb2fe412639227f; no voters:
I20260620 00:15:09.729488 14649 raft_consensus.cc:2781] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:09.729713 14649 raft_consensus.cc:686] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 LEADER]: Becoming Leader. State: Replica: 0624e3bcc7a94ad39f852a8fd35ef7cd, State: Running, Role: LEADER
I20260620 00:15:09.729836 14649 consensus_queue.cc:237] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:09.730753 14166 catalog_manager.cc:5495] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } health_report { overall_health: UNKNOWN } } }
I20260620 00:15:09.750178 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.747581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.0.98.1:37443)
0620 00:15:09.747662 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:09.747664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:09.747678 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:15:09.747715 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:09.747716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:09.747737 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:09.747829 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:09.747990 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.748461 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.749330 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:09.749531 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:09.749595 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:09.749772 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:09.749889 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:09.749955 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:09.750002 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 00:15:09.750032 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:15:09.752295 14445 raft_consensus.cc:1261] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Refusing update from remote peer 0624e3bcc7a94ad39f852a8fd35ef7cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:09.752295 14579 raft_consensus.cc:1261] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Refusing update from remote peer 0624e3bcc7a94ad39f852a8fd35ef7cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:09.752764 14649 consensus_queue.cc:1035] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:09.752938 14652 consensus_queue.cc:1035] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:09.755501 14655 mvcc.cc:205] Tried to move back new op lower bound from 7298721831941545984 to 7298721831854055424. Current Snapshot: MvccSnapshot[applied={T|T < 7298721831941545984}]
I20260620 00:15:09.756512 14656 mvcc.cc:205] Tried to move back new op lower bound from 7298721831941545984 to 7298721831854055424. Current Snapshot: MvccSnapshot[applied={T|T < 7298721831941545984}]
I20260620 00:15:09.758595 14165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:09.758719 14165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:09.759606 14165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Sending DeleteTablet for 3 replicas of tablet 59178bf341c54ea991631b85e8707c3d
I20260620 00:15:09.760058 14425 tablet_service.cc:1514] Processing DeleteTablet for tablet 59178bf341c54ea991631b85e8707c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49994
I20260620 00:15:09.760106 14559 tablet_service.cc:1514] Processing DeleteTablet for tablet 59178bf341c54ea991631b85e8707c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60620
I20260620 00:15:09.760087 14291 tablet_service.cc:1514] Processing DeleteTablet for tablet 59178bf341c54ea991631b85e8707c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43056
I20260620 00:15:09.760356 14664 tablet_replica.cc:330] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: stopping tablet replica
I20260620 00:15:09.760370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14228
I20260620 00:15:09.762027 14666 tablet_replica.cc:330] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: stopping tablet replica
I20260620 00:15:09.762189 14666 raft_consensus.cc:2229] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:09.762349 14666 raft_consensus.cc:2258] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:09.763164 14664 raft_consensus.cc:2229] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:09.763352 14664 raft_consensus.cc:2258] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:09.763927 14664 ts_tablet_manager.cc:1905] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:09.765057 14666 ts_tablet_manager.cc:1905] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:09.765942 14664 ts_tablet_manager.cc:1918] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:09.766115 14664 log.cc:1196] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/59178bf341c54ea991631b85e8707c3d
W20260620 00:15:09.766132 14150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37443 (error 108)
I20260620 00:15:09.766484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14362
I20260620 00:15:09.766552 14664 ts_tablet_manager.cc:1939] T 59178bf341c54ea991631b85e8707c3d P 457fab3163fc45938521f784ca461eff: Deleting consensus metadata
I20260620 00:15:09.766841 14666 ts_tablet_manager.cc:1918] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:09.766923 14666 log.cc:1196] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/59178bf341c54ea991631b85e8707c3d
I20260620 00:15:09.769320 14150 catalog_manager.cc:4841] TS 457fab3163fc45938521f784ca461eff (127.0.98.3:42711): tablet 59178bf341c54ea991631b85e8707c3d (table test-table [id=b977c4e9b3a4418ea467789918a848a5]) successfully deleted
I20260620 00:15:09.769515 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.769252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37443 (local address 127.0.0.1:43086)
0620 00:15:09.769325 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.769445 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 00:15:09.769681 14150 catalog_manager.cc:4584] TS 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59178bf341c54ea991631b85e8707c3d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37443: connect: Connection refused (error 111)
I20260620 00:15:09.772972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14496
I20260620 00:15:09.773049 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:09.772888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38573 (local address 127.0.0.1:50002)
0620 00:15:09.772955 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:09.772985 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:15:09.778445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14134
I20260620 00:15:09.784309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=36367
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:09.869621 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:09.869843 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:09.869867 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:09.871107 14668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:09.871158 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:09.871172 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:09.871186 14668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:09.871248 14668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:09.872864 14668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:09.873129 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:09.873386 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:09.882620 14675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.882622 14674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:09.882622 14677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:09.882861 14668 server_base.cc:1034] running on GCE node
I20260620 00:15:09.883210 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:09.883661 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:09.884871 14668 hybrid_clock.cc:648] HybridClock initialized: now 1781914509884803 us; error 64 us; skew 500 ppm
Jun 20 00:15:09 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:09.893246 14668 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:09.893824 14668 webserver.cc:460] Webserver started at http://127.0.98.62:36367/ using document root <none> and password file <none>
I20260620 00:15:09.894094 14668 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:09.894182 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:09.895768 14668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:09.896613 14684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:09.896813 14668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:09.896938 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "f9c21907f3974680bc1c88153873c19e"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "554b71941b40aa0e16cfba7c6a26025f"
server_key_iv: "12c95641b493eaf3650cf3bd3295b8be"
server_key_version: "encryptionkey@0"
I20260620 00:15:09.897385 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:09.912286 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:09.914407 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:09.914588 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:09.919155 14668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260620 00:15:09.919195 14736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260620 00:15:09.919622 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:09.919914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14668
I20260620 00:15:09.920238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=37951
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:09.921492 14737 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:15:09.922361 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap starting.
I20260620 00:15:09.934125 14737 log.cc:824] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:09.936988 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:09.937415 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap complete.
I20260620 00:15:09.939950 14737 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.940220 14737 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Initialized, Role: FOLLOWER
I20260620 00:15:09.940392 14737 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.940476 14737 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:09.940514 14737 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:09.940609 14737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:15:09.942128 14737 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.942363 14737 leader_election.cc:302] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c21907f3974680bc1c88153873c19e; no voters:
I20260620 00:15:09.942605 14737 leader_election.cc:288] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:15:09.942668 14741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:15:09.942966 14741 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 LEADER]: Becoming Leader. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Running, Role: LEADER
I20260620 00:15:09.943038 14737 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:09.943079 14741 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:09.943387 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c21907f3974680bc1c88153873c19e. Latest consensus state: current_term: 2 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:09.943460 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:09.943604 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:09.943681 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:09.943979 14746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:09.944903 14746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:15:09.945034 14746 catalog_manager.cc:1259] Loaded cluster ID: 769f2a177a574dbcb158337ab57d7b19
I20260620 00:15:09.945073 14746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:09.945353 14746 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 00:15:09.950336 14746 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 00:15:09.953214 14746 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:09.953469 14746 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Loaded TSK: 0
I20260620 00:15:09.953773 14746 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:15:10.018440 14739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.018915 14739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.019032 14739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.020606 14739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.020691 14739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.020742 14739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:10.022595 14739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37951
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.022921 14739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.023206 14739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.028991 14764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.029229 14767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.028988 14765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.029655 14739 server_base.cc:1034] running on GCE node
I20260620 00:15:10.029954 14739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.030272 14739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:10.031471 14739 hybrid_clock.cc:648] HybridClock initialized: now 1781914510031439 us; error 44 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:10.035012 14739 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:10.035609 14739 webserver.cc:460] Webserver started at http://127.0.98.1:37951/ using document root <none> and password file <none>
I20260620 00:15:10.035861 14739 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:10.035954 14739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:10.037762 14739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:10.038648 14774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.038897 14739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:10.039011 14739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b863d328275a145a5740cbb3e93c90d4"
server_key_iv: "064eda5384dabee6bddfc737db293cc0"
server_key_version: "encryptionkey@0"
I20260620 00:15:10.039480 14739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:10.049348 14739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:10.051857 14739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:10.052037 14739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:10.052313 14739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:10.052871 14781 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:15:10.054998 14739 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:15:10.055114 14739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:15:10.055173 14739 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:15:10.056161 14739 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:15:10.056242 14739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:10.056308 14781 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Bootstrap starting.
I20260620 00:15:10.064396 14739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37443
I20260620 00:15:10.064441 14888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37443 every 8 connection(s)
I20260620 00:15:10.064929 14739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:15:10.066659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14739
I20260620 00:15:10.067051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38573
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=39727
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:10.070781 14781 log.cc:824] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:10.074141 14781 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:10.074484 14781 tablet_bootstrap.cc:492] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Bootstrap complete.
I20260620 00:15:10.075752 14781 ts_tablet_manager.cc:1397] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.011s
I20260620 00:15:10.077318 14781 raft_consensus.cc:348] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:10.077674 14781 raft_consensus.cc:729] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0624e3bcc7a94ad39f852a8fd35ef7cd, State: Initialized, Role: FOLLOWER
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:10.077860 14781 consensus_queue.cc:260] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } } peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } }
I20260620 00:15:10.078189 14781 ts_tablet_manager.cc:1428] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
I20260620 00:15:10.083838 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.065770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:52739)
0620 00:15:10.065964 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.066010 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.066633 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.069400 (+ 2767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.069406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.069833 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:10.070051 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.070058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.072105 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.072106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.073908 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.073912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.075000 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.075002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.075196 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.075986 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:10.075998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:10.078439 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:10.081515 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.081526 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.081529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.081938 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.082334 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.082335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.082335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.082499 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.083092 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:10.083099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:10.083285 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.083381 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.083579 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260620 00:15:10.084683 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.065893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52739 (local address 127.0.98.62:43019)
0620 00:15:10.068845 (+ 2952us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.068858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.068910 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:15:10.069085 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.069094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.069162 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.070171 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:10.070437 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.071927 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.074077 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.074840 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.075609 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.078602 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:10.078607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:10.078608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:10.078627 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:10.081373 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.082066 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.082070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.082071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.082214 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.082600 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.082601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.082602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.082988 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:10.084180 (+ 1192us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.084275 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.084437 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2884,"thread_start_us":102,"threads_started":1}
I20260620 00:15:10.085597 14889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:10.085708 14889 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:10.085918 14889 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:10.086824 14701 ts_manager.cc:194] Registered new tserver with Master: 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443)
I20260620 00:15:10.088235 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52739
I20260620 00:15:10.089463 14889 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:15:10.099593 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.087556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37443 (local address 127.0.0.1:43100)
0620 00:15:10.088026 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.088067 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.088213 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.088971 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.088977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.089330 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:10.089393 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.089396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.090983 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.090985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.092037 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.092041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.092809 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.092814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.092904 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.093488 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:10.093495 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:10.095505 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:10.098046 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.098056 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.098059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.098386 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.098680 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.098681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.098681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.098758 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.099125 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:10.099133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:10.099202 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.099324 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.099477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":105,"threads_started":1}
I20260620 00:15:10.099602 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.087851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.0.98.1:37443)
0620 00:15:10.088416 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.088422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.088436 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:15:10.088489 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.088491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.088508 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.088898 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:10.089492 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.090876 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.092149 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.092720 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.092801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.095630 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:10.095644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:10.095645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:10.095655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:10.097921 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.098502 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.098506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.098506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.098581 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.098831 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.098832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.098832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.099039 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:10.099113 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.099385 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.099498 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":79,"threads_started":1}
I20260620 00:15:10.101208 14823 tablet_service.cc:1514] Processing DeleteTablet for tablet 59178bf341c54ea991631b85e8707c3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43100
I20260620 00:15:10.101966 14898 tablet_replica.cc:330] stopping tablet replica
I20260620 00:15:10.102083 14898 raft_consensus.cc:2229] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:10.102154 14898 raft_consensus.cc:2258] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:10.102525 14898 ts_tablet_manager.cc:1905] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.105366 14898 ts_tablet_manager.cc:1918] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:10.105468 14898 log.cc:1196] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/59178bf341c54ea991631b85e8707c3d
I20260620 00:15:10.105729 14898 ts_tablet_manager.cc:1939] T 59178bf341c54ea991631b85e8707c3d P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting consensus metadata
I20260620 00:15:10.106503 14685 catalog_manager.cc:4841] TS 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443): tablet 59178bf341c54ea991631b85e8707c3d (table test-table [id=b977c4e9b3a4418ea467789918a848a5]) successfully deleted
W20260620 00:15:10.167315 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.167577 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.167623 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.169876 14893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.169968 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.170018 14893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:10.172451 14893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39727
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.172773 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.173035 14893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.178781 14906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.178757 14903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.178757 14904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.178918 14893 server_base.cc:1034] running on GCE node
I20260620 00:15:10.179275 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.179562 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:10.180732 14893 hybrid_clock.cc:648] HybridClock initialized: now 1781914510180726 us; error 43 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:10.183561 14893 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:10.184077 14893 webserver.cc:460] Webserver started at http://127.0.98.2:39727/ using document root <none> and password file <none>
I20260620 00:15:10.184334 14893 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:10.184397 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:10.185976 14893 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:10.186741 14913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.186901 14893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:15:10.187011 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "564fa8d4587943c48cb2fe412639227f"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b666aff3d55c3f66676fcf929474c567"
server_key_iv: "33688b393db46c589499b7717aa2cbe2"
server_key_version: "encryptionkey@0"
I20260620 00:15:10.187427 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:10.206477 14893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:10.208576 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:10.208741 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:10.208998 14893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:10.209482 14920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 00:15:10.211493 14920 ts_tablet_manager.cc:1852] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 00:15:10.211901 14920 ts_tablet_manager.cc:1905] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.214566 14920 ts_tablet_manager.cc:1918] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:10.214664 14920 log.cc:1196] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/59178bf341c54ea991631b85e8707c3d
I20260620 00:15:10.214902 14920 ts_tablet_manager.cc:1939] T 59178bf341c54ea991631b85e8707c3d P 564fa8d4587943c48cb2fe412639227f: Deleting consensus metadata
I20260620 00:15:10.215090 14893 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 00:15:10.215133 14893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 00:15:10.215176 14893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:10.215194 14893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.221449 14893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38573
I20260620 00:15:10.221568 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38573 every 8 connection(s)
I20260620 00:15:10.221851 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:15:10.225716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 14893
I20260620 00:15:10.226033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42711
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=45119
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:10.231987 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.222610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55005 (local address 127.0.98.62:43019)
0620 00:15:10.222661 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.222663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.222873 (+ 210us) server_negotiation.cc:407] Connection header received
0620 00:15:10.223497 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.223499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.223519 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.223584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:10.224236 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.224813 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.226050 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.226343 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.226437 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.228964 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:10.228966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:10.228966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:10.228975 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:10.230476 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.230927 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.230929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.230929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.230993 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.231254 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.231255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.231255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.231411 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:10.231490 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.231792 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.231888 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:15:10.232214 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.222571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:55005)
0620 00:15:10.222725 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.222778 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.223381 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.223655 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.223659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.223929 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:10.224128 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.224131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.224911 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.224912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.225964 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.225967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.226466 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.226467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.226630 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.227181 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:10.227192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:10.228860 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:10.230587 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.230595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.230598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.230838 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.231062 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.231063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.231063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.231185 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.231491 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:10.231499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:10.231656 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.231771 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.231953 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260620 00:15:10.232779 15028 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:10.232900 15028 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:10.233119 15028 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:10.233589 14701 ts_manager.cc:194] Registered new tserver with Master: 564fa8d4587943c48cb2fe412639227f (127.0.98.2:38573)
I20260620 00:15:10.233983 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55005
W20260620 00:15:10.311367 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.311718 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.311766 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.313508 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.313596 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.313647 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:10.315436 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45119
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.315778 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.316052 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.322650 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.322688 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.322886 15031 server_base.cc:1034] running on GCE node
W20260620 00:15:10.323060 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.323318 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.323697 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:10.324898 15031 hybrid_clock.cc:648] HybridClock initialized: now 1781914510324877 us; error 34 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:10.327781 15031 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:10.328420 15031 webserver.cc:460] Webserver started at http://127.0.98.3:45119/ using document root <none> and password file <none>
I20260620 00:15:10.328751 15031 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:10.328838 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:10.330941 15031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:15:10.332089 15046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.332430 15031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:10.332718 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "457fab3163fc45938521f784ca461eff"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "7d568996395747ace8ee2263faec58d6"
server_key_iv: "6c436262dee14f84ee25e05cac1c3f25"
server_key_version: "encryptionkey@0"
I20260620 00:15:10.333165 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:10.346269 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:10.348743 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:10.348951 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:10.349221 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:10.349557 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:10.349617 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.349661 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:10.349713 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.357436 15031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42711
I20260620 00:15:10.357582 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42711 every 8 connection(s)
I20260620 00:15:10.357986 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:15:10.361795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15031
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:10.371098 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.362412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36850 (local address 127.0.98.62:43019)
0620 00:15:10.362601 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.362603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.362611 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:15:10.362635 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.362636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.362649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.362721 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:10.363469 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.363972 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.364887 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.365105 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.365194 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.367450 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:10.367451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:10.367452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:10.367460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:10.369589 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.370092 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.370094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.370094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.370193 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.370469 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.370470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.370470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.370639 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:10.370848 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.370950 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.371021 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260620 00:15:10.372736 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.359062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33915 (local address 127.0.98.62:43019)
0620 00:15:10.359114 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.359116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.359434 (+ 318us) server_negotiation.cc:407] Connection header received
0620 00:15:10.360693 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.360696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.360755 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.360833 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:10.362349 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.363085 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.364969 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.365333 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.365418 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.368766 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:10.368768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:10.368768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:10.368778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:10.370639 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.371527 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.371528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.371528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.371595 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:10.371918 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:10.371919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:10.371920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:10.372087 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:10.372146 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.372504 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.372644 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":24}
I20260620 00:15:10.372967 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.358865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:33915)
0620 00:15:10.359157 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.359207 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.360456 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.360960 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.360965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.361891 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:10.362232 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.362239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.363213 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.363214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.364848 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.364853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.365443 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.365444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.365618 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.366200 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:10.366212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:10.368672 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:10.370754 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.370765 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.370768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.371169 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.371662 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:10.371663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:10.371663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:10.371859 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:10.372151 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:10.372159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:10.372347 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.372466 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.372678 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260620 00:15:10.373880 15160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43019
I20260620 00:15:10.374171 15160 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:10.374482 15160 heartbeater.cc:511] Master 127.0.98.62:43019 requested a full tablet report, sending...
I20260620 00:15:10.375186 14701 ts_manager.cc:194] Registered new tserver with Master: 457fab3163fc45938521f784ca461eff (127.0.98.3:42711)
I20260620 00:15:10.375576 14701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33915
I20260620 00:15:10.383471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:15:10.386684 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.383888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.98.62:43019)
0620 00:15:10.383970 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.383972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.384033 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:15:10.384096 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.384096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.384134 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.384198 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:10.384394 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.385056 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.385837 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.386039 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.386102 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.386275 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:10.386389 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:10.386463 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.386529 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.386547 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:15:10.387068 14701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:15:10.389143 14701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:15:10.391927 14823 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 (DEFAULT_TABLE table=test-table [id=6ecd8a259168451a94ed355d6ef78b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:10.392151 14823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:10.393841 14781 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Bootstrap starting.
I20260620 00:15:10.394301 14781 tablet_bootstrap.cc:654] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:10.394958 14781 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: No bootstrap required, opened a new log
I20260620 00:15:10.395013 14781 ts_tablet_manager.cc:1397] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:10.395406 14781 raft_consensus.cc:348] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.395525 14781 raft_consensus.cc:374] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:10.395596 14781 raft_consensus.cc:729] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0624e3bcc7a94ad39f852a8fd35ef7cd, State: Initialized, Role: FOLLOWER
I20260620 00:15:10.395655 14781 consensus_queue.cc:260] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.395767 14781 ts_tablet_manager.cc:1428] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:10.395967 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.391382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.0.1:60624)
0620 00:15:10.391506 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.391525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.391592 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.391886 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.391887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.391889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:10.391944 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.391946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.393336 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.393337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.394403 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.394405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.395709 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.395710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.395787 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.395792 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.395834 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.395889 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 00:15:10.395983 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.391749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.2:38573)
0620 00:15:10.392098 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.392102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.392113 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:15:10.392149 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.392150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.392167 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.392330 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:10.392529 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.393561 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.394875 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.395457 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.395554 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.395621 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.395861 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.395902 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":85,"threads_started":1}
I20260620 00:15:10.396168 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.391542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.98.3:42711)
0620 00:15:10.391698 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.391703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.391715 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:15:10.391758 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.391761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.391779 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.391872 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:10.392043 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.393195 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.394500 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.395475 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.395717 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.395866 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.395897 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.395938 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260620 00:15:10.396499 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.391382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38573 (local address 127.0.0.1:50010)
0620 00:15:10.391613 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.391623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.391667 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.392324 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.392325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.392328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:10.392439 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.392442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.393658 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.393659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.394780 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.394783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.395575 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.395576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.395682 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.395705 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.396154 (+ 449us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.396200 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":21,"thread_start_us":143,"threads_started":1}
I20260620 00:15:10.396950 15094 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 (DEFAULT_TABLE table=test-table [id=6ecd8a259168451a94ed355d6ef78b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:10.397243 14962 tablet_service.cc:1467] Processing CreateTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 (DEFAULT_TABLE table=test-table [id=6ecd8a259168451a94ed355d6ef78b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:10.397341 15094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:10.397534 14962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:10.399947 14920 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Bootstrap starting.
I20260620 00:15:10.400866 14920 tablet_bootstrap.cc:654] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:10.401240 14920 log.cc:824] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:10.401510 15170 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Bootstrap starting.
I20260620 00:15:10.402083 14920 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: No bootstrap required, opened a new log
I20260620 00:15:10.402151 14920 ts_tablet_manager.cc:1397] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:15:10.402295 15170 tablet_bootstrap.cc:654] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:10.402933 15170 log.cc:824] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:10.404184 15170 tablet_bootstrap.cc:492] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: No bootstrap required, opened a new log
I20260620 00:15:10.404285 15170 ts_tablet_manager.cc:1397] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:15:10.404441 14920 raft_consensus.cc:348] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.404657 14920 raft_consensus.cc:374] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:10.404685 14920 raft_consensus.cc:729] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564fa8d4587943c48cb2fe412639227f, State: Initialized, Role: FOLLOWER
I20260620 00:15:10.404814 14920 consensus_queue.cc:260] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.405102 15028 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
I20260620 00:15:10.405112 14920 ts_tablet_manager.cc:1428] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 00:15:10.406901 15170 raft_consensus.cc:348] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.407136 15170 raft_consensus.cc:374] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:10.407178 15170 raft_consensus.cc:729] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457fab3163fc45938521f784ca461eff, State: Initialized, Role: FOLLOWER
I20260620 00:15:10.407315 15170 consensus_queue.cc:260] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.407656 15170 ts_tablet_manager.cc:1428] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:15:10.407693 15160 heartbeater.cc:503] Master 127.0.98.62:43019 was elected leader, sending a full tablet report...
W20260620 00:15:10.472473 15029 tablet.cc:2369] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:10.480633 15173 raft_consensus.cc:482] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:10.480768 15173 raft_consensus.cc:504] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.481077 15173 leader_election.cc:288] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 457fab3163fc45938521f784ca461eff (127.0.98.3:42711), 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443)
I20260620 00:15:10.485292 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.481460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42411 (local address 127.0.98.3:42711)
0620 00:15:10.481530 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.481532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.481550 (+ 18us) server_negotiation.cc:407] Connection header received
0620 00:15:10.481599 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.481600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.481613 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.481702 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:10.481932 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.482840 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.484325 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.484886 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.484988 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.484999 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.485171 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.485209 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:15:10.485293 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.481688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60891 (local address 127.0.98.1:37443)
0620 00:15:10.481748 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.481750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.481902 (+ 152us) server_negotiation.cc:407] Connection header received
0620 00:15:10.481939 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.481940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.481967 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.482045 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:10.482169 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.482992 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.484359 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.484959 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.485048 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.485127 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.485179 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.485217 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:15:10.485347 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.481346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42711 (local address 127.0.98.2:42411)
0620 00:15:10.481414 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.481430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.481508 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.481718 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.481719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.481721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:10.481822 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.481824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.482953 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.482954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.484219 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.484222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.484991 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.484992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.485089 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.485092 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.485164 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.485209 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:15:10.485345 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.481575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37443 (local address 127.0.98.2:60891)
0620 00:15:10.481824 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.481835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.481888 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.482055 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:10.482056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:10.482057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:10.482104 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.482106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.483087 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.483088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.484221 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:10.484222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.485050 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.485051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:10.485120 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.485124 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:10.485174 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:15:10.485216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2,"thread_start_us":160,"threads_started":1}
I20260620 00:15:10.485783 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccd2fcd1ae4f568a9f0c0bbaeb7174" candidate_uuid: "564fa8d4587943c48cb2fe412639227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457fab3163fc45938521f784ca461eff" is_pre_election: true
I20260620 00:15:10.485819 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccd2fcd1ae4f568a9f0c0bbaeb7174" candidate_uuid: "564fa8d4587943c48cb2fe412639227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" is_pre_election: true
I20260620 00:15:10.485956 15114 raft_consensus.cc:2454] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564fa8d4587943c48cb2fe412639227f in term 0.
I20260620 00:15:10.485956 14843 raft_consensus.cc:2454] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 564fa8d4587943c48cb2fe412639227f in term 0.
I20260620 00:15:10.486176 14914 leader_election.cc:302] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 457fab3163fc45938521f784ca461eff, 564fa8d4587943c48cb2fe412639227f; no voters:
I20260620 00:15:10.486320 15173 raft_consensus.cc:2781] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:10.486375 15173 raft_consensus.cc:482] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:10.486405 15173 raft_consensus.cc:3037] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:10.487079 15173 raft_consensus.cc:504] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.487216 15173 leader_election.cc:288] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [CANDIDATE]: Term 1 election: Requested vote from peers 457fab3163fc45938521f784ca461eff (127.0.98.3:42711), 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443)
I20260620 00:15:10.487401 14843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccd2fcd1ae4f568a9f0c0bbaeb7174" candidate_uuid: "564fa8d4587943c48cb2fe412639227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
I20260620 00:15:10.487416 15114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dccd2fcd1ae4f568a9f0c0bbaeb7174" candidate_uuid: "564fa8d4587943c48cb2fe412639227f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "457fab3163fc45938521f784ca461eff"
I20260620 00:15:10.487485 14843 raft_consensus.cc:3037] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:10.487485 15114 raft_consensus.cc:3037] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:10.488317 15114 raft_consensus.cc:2454] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564fa8d4587943c48cb2fe412639227f in term 1.
I20260620 00:15:10.488317 14843 raft_consensus.cc:2454] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 564fa8d4587943c48cb2fe412639227f in term 1.
I20260620 00:15:10.488503 14914 leader_election.cc:302] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0624e3bcc7a94ad39f852a8fd35ef7cd, 564fa8d4587943c48cb2fe412639227f; no voters:
I20260620 00:15:10.488613 15173 raft_consensus.cc:2781] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:10.488770 15173 raft_consensus.cc:686] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 1 LEADER]: Becoming Leader. State: Replica: 564fa8d4587943c48cb2fe412639227f, State: Running, Role: LEADER
I20260620 00:15:10.488873 15173 consensus_queue.cc:237] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } }
I20260620 00:15:10.489837 14701 catalog_manager.cc:5495] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f reported cstate change: term changed from 0 to 1, leader changed from <none> to 564fa8d4587943c48cb2fe412639227f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "564fa8d4587943c48cb2fe412639227f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "564fa8d4587943c48cb2fe412639227f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 } health_report { overall_health: UNKNOWN } } }
I20260620 00:15:10.513598 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.510809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.0.98.2:38573)
0620 00:15:10.510877 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:10.510879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:10.510909 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:15:10.511051 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:10.511052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:10.511073 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:10.511147 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:10.511275 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.511742 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.512628 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:10.512856 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:10.512919 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:10.513113 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:10.513265 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:10.513349 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:10.513387 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:15:10.513410 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:15:10.515573 15114 raft_consensus.cc:1261] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Refusing update from remote peer 564fa8d4587943c48cb2fe412639227f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:10.515630 14843 raft_consensus.cc:1261] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Refusing update from remote peer 564fa8d4587943c48cb2fe412639227f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:10.515992 15176 consensus_queue.cc:1035] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [LEADER]: Connected to new peer: Peer: permanent_uuid: "457fab3163fc45938521f784ca461eff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:10.516147 15173 consensus_queue.cc:1035] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:10.518844 15179 mvcc.cc:205] Tried to move back new op lower bound from 7298721835069046784 to 7298721834963116032. Current Snapshot: MvccSnapshot[applied={T|T < 7298721835069046784}]
I20260620 00:15:10.519956 15181 mvcc.cc:205] Tried to move back new op lower bound from 7298721835069046784 to 7298721834963116032. Current Snapshot: MvccSnapshot[applied={T|T < 7298721835069046784 or (T in {7298721835069046784})}]
I20260620 00:15:10.522306 14700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:10.522439 14700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:10.523532 14700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Sending DeleteTablet for 3 replicas of tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174
I20260620 00:15:10.524123 14823 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43100
I20260620 00:15:10.524152 15094 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60624
I20260620 00:15:10.524137 14962 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260620 00:15:10.524240 14898 tablet_replica.cc:330] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: stopping tablet replica
I20260620 00:15:10.524291 14898 raft_consensus.cc:2229] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:10.524362 14898 raft_consensus.cc:2258] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:10.524598 15190 tablet_replica.cc:330] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: stopping tablet replica
I20260620 00:15:10.524722 15189 tablet_replica.cc:330] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: stopping tablet replica
I20260620 00:15:10.524721 15190 raft_consensus.cc:2229] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:15:10.524820 14898 ts_tablet_manager.cc:1905] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.524881 15190 raft_consensus.cc:2258] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:10.524931 15189 raft_consensus.cc:2229] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:10.525151 15189 raft_consensus.cc:2258] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:10.525596 15189 ts_tablet_manager.cc:1905] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.526470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14739
I20260620 00:15:10.527340 15189 ts_tablet_manager.cc:1918] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:10.527437 15189 log.cc:1196] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/2dccd2fcd1ae4f568a9f0c0bbaeb7174
I20260620 00:15:10.527720 15189 ts_tablet_manager.cc:1939] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 457fab3163fc45938521f784ca461eff: Deleting consensus metadata
I20260620 00:15:10.528240 14685 catalog_manager.cc:4841] TS 457fab3163fc45938521f784ca461eff (127.0.98.3:42711): tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 (table test-table [id=6ecd8a259168451a94ed355d6ef78b22]) successfully deleted
I20260620 00:15:10.529479 15190 ts_tablet_manager.cc:1905] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.531733 15190 ts_tablet_manager.cc:1918] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:10.531817 15190 log.cc:1196] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/2dccd2fcd1ae4f568a9f0c0bbaeb7174
I20260620 00:15:10.532099 15190 ts_tablet_manager.cc:1939] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 564fa8d4587943c48cb2fe412639227f: Deleting consensus metadata
I20260620 00:15:10.532802 14688 catalog_manager.cc:4841] TS 564fa8d4587943c48cb2fe412639227f (127.0.98.2:38573): tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174 (table test-table [id=6ecd8a259168451a94ed355d6ef78b22]) successfully deleted
W20260620 00:15:10.533887 14685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37443 (error 108)
I20260620 00:15:10.534189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14893
I20260620 00:15:10.535125 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.534315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37443 (local address 127.0.0.1:43104)
0620 00:15:10.535050 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.535095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":709}
W20260620 00:15:10.535434 14685 catalog_manager.cc:4584] TS 0624e3bcc7a94ad39f852a8fd35ef7cd (127.0.98.1:37443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dccd2fcd1ae4f568a9f0c0bbaeb7174: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37443: connect: Connection refused (error 111)
I20260620 00:15:10.540938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15031
I20260620 00:15:10.546139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 14668
I20260620 00:15:10.552209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=36367
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:10.636194 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.636608 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.636665 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.638280 15192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:10.638362 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.638396 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.638428 15192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:10.638458 15192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:10.640271 15192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.640679 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.640988 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.647168 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.647255 15192 server_base.cc:1034] running on GCE node
W20260620 00:15:10.647169 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.647583 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.647902 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.648253 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:10.649471 15192 hybrid_clock.cc:648] HybridClock initialized: now 1781914510649404 us; error 44 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:10.652496 15192 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:10.653326 15192 webserver.cc:460] Webserver started at http://127.0.98.62:36367/ using document root <none> and password file <none>
I20260620 00:15:10.653739 15192 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:10.653857 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:10.655426 15192 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:15:10.656255 15207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.656654 15192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:10.656780 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "f9c21907f3974680bc1c88153873c19e"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "554b71941b40aa0e16cfba7c6a26025f"
server_key_iv: "12c95641b493eaf3650cf3bd3295b8be"
server_key_version: "encryptionkey@0"
I20260620 00:15:10.657191 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:10.670902 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:10.673472 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:10.673650 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:10.677995 15192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260620 00:15:10.678030 15259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260620 00:15:10.678437 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:10.680114 15260 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:15:10.680895 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap starting.
I20260620 00:15:10.687942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15192
I20260620 00:15:10.688333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=37951
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:10.690003 15260 log.cc:824] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:10.692763 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:10.693080 15260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e: Bootstrap complete.
I20260620 00:15:10.695758 15260 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:10.696038 15260 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Initialized, Role: FOLLOWER
I20260620 00:15:10.696250 15260 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:10.696362 15260 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:10.696424 15260 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:10.696473 15260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:15:10.697587 15260 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:10.697726 15260 leader_election.cc:302] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c21907f3974680bc1c88153873c19e; no voters:
I20260620 00:15:10.697938 15260 leader_election.cc:288] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:15:10.697980 15264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:15:10.698096 15264 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [term 3 LEADER]: Becoming Leader. State: Replica: f9c21907f3974680bc1c88153873c19e, State: Running, Role: LEADER
I20260620 00:15:10.698189 15264 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260620 00:15:10.698232 15260 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:10.698465 15265 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:10.698555 15265 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:10.698458 15264 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c21907f3974680bc1c88153873c19e. Latest consensus state: current_term: 3 leader_uuid: "f9c21907f3974680bc1c88153873c19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c21907f3974680bc1c88153873c19e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260620 00:15:10.698746 15264 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c21907f3974680bc1c88153873c19e [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:10.699327 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:10.700165 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:15:10.700270 15268 catalog_manager.cc:1259] Loaded cluster ID: 769f2a177a574dbcb158337ab57d7b19
I20260620 00:15:10.700294 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 00:15:10.700531 15268 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffb013299ad google::LogMessage::Fail() at ??:0
@ 0x7ffb0132c88a google::LogMessage::SendToLog() at ??:0
@ 0x7ffb013294de google::LogMessage::Flush() at ??:0
@ 0x7ffb0132aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffb0924e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ffb092578b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffb031f70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffb031ec954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffb040506db start_thread at ??:0
@ 0x7ffaffe2761f clone at ??:0
W20260620 00:15:10.814953 15262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.815207 15262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.815250 15262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.816728 15262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.816787 15262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.816819 15262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:10.818545 15262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37951
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.818832 15262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.819110 15262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.824759 15296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.824791 15297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.824753 15299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.825284 15262 server_base.cc:1034] running on GCE node
I20260620 00:15:10.825568 15262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.825870 15262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:10.827093 15262 hybrid_clock.cc:648] HybridClock initialized: now 1781914510827066 us; error 45 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:10.829887 15262 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:10.830502 15262 webserver.cc:460] Webserver started at http://127.0.98.1:37951/ using document root <none> and password file <none>
I20260620 00:15:10.830773 15262 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:10.830849 15262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:10.832501 15262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:15:10.833454 15306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.833664 15262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:10.833782 15262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "0624e3bcc7a94ad39f852a8fd35ef7cd"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b863d328275a145a5740cbb3e93c90d4"
server_key_iv: "064eda5384dabee6bddfc737db293cc0"
server_key_version: "encryptionkey@0"
I20260620 00:15:10.834240 15262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:10.867686 15262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:10.870397 15262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:10.870589 15262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:10.870852 15262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:10.871369 15313 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 00:15:10.873569 15313 ts_tablet_manager.cc:1852] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 00:15:10.873992 15313 ts_tablet_manager.cc:1905] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:10.876904 15313 ts_tablet_manager.cc:1918] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:10.877022 15313 log.cc:1196] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/2dccd2fcd1ae4f568a9f0c0bbaeb7174
I20260620 00:15:10.877328 15313 ts_tablet_manager.cc:1939] T 2dccd2fcd1ae4f568a9f0c0bbaeb7174 P 0624e3bcc7a94ad39f852a8fd35ef7cd: Deleting consensus metadata
I20260620 00:15:10.877657 15262 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 00:15:10.877785 15262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260620 00:15:10.877854 15262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:10.877878 15262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:10.885677 15262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37443
I20260620 00:15:10.885812 15420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37443 every 8 connection(s)
I20260620 00:15:10.886169 15262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:15:10.895855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15262
I20260620 00:15:10.896188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38573
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=39727
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:10.950696 15423 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:15:10.887000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:37781)
0620 00:15:10.887227 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.887315 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:10.888186 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:10.950441 (+ 62255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260620 00:15:10.950968 15421 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43019 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43019: Client connection negotiation failed: client connection to 127.0.98.62:43019: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:15:10.951366 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.951183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:40505)
0620 00:15:10.951295 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.951345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 00:15:10.951835 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:10.951695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:40079)
0620 00:15:10.951779 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:10.951813 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 00:15:10.951990 15421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:15:10.988034 15424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:10.988410 15424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:10.988461 15424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:10.989959 15424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:10.990051 15424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:10.990103 15424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:10.991868 15424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39727
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:10.992210 15424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:10.992489 15424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:10.998533 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.998492 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:10.998555 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:10.998772 15424 server_base.cc:1034] running on GCE node
I20260620 00:15:10.999032 15424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:10.999347 15424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:11.000603 15424 hybrid_clock.cc:648] HybridClock initialized: now 1781914511000578 us; error 40 us; skew 500 ppm
Jun 20 00:15:10 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:11.003818 15424 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:11.004643 15424 webserver.cc:460] Webserver started at http://127.0.98.2:39727/ using document root <none> and password file <none>
I20260620 00:15:11.004923 15424 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:11.005004 15424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:11.006876 15424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:15:11.008049 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.008517 15424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:11.008759 15424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "564fa8d4587943c48cb2fe412639227f"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "b666aff3d55c3f66676fcf929474c567"
server_key_iv: "33688b393db46c589499b7717aa2cbe2"
server_key_version: "encryptionkey@0"
I20260620 00:15:11.009227 15424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:11.032773 15424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:11.036132 15424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:11.036389 15424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:11.036937 15424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:11.037442 15424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:11.037513 15424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.037557 15424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:11.037587 15424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.044090 15424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38573
I20260620 00:15:11.044155 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38573 every 8 connection(s)
I20260620 00:15:11.044658 15424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:15:11.046128 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.045656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:52545)
0620 00:15:11.045873 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.045968 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
W20260620 00:15:11.046697 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43019 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43019: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
I20260620 00:15:11.047070 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.046967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:46137)
0620 00:15:11.047015 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.047049 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 00:15:11.047446 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.047341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:59179)
0620 00:15:11.047393 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.047430 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:15:11.047608 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:15:11.053015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15424
I20260620 00:15:11.053598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42711
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43019
--webserver_port=45119
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:11.147845 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:11.148114 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:11.148144 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:11.149580 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:11.149638 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:11.149669 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:11.151410 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45119
--tserver_master_addrs=127.0.98.62:43019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:11.151723 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:11.152001 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:11.157563 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:11.157557 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:11.157563 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:11.157920 15556 server_base.cc:1034] running on GCE node
I20260620 00:15:11.158185 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:11.158481 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:11.159657 15556 hybrid_clock.cc:648] HybridClock initialized: now 1781914511159640 us; error 29 us; skew 500 ppm
Jun 20 00:15:11 dist-test-slave-28bv krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:11.162426 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:11.162938 15556 webserver.cc:460] Webserver started at http://127.0.98.3:45119/ using document root <none> and password file <none>
I20260620 00:15:11.163187 15556 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:11.163236 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:11.164710 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:11.165398 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.165617 15556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:11.165692 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "457fab3163fc45938521f784ca461eff"
format_stamp: "Formatted at 2026-06-20 00:15:09 on dist-test-slave-28bv"
server_key: "7d568996395747ace8ee2263faec58d6"
server_key_iv: "6c436262dee14f84ee25e05cac1c3f25"
server_key_version: "encryptionkey@0"
I20260620 00:15:11.166088 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:11.185020 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:11.187418 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:11.187567 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:11.187839 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:11.188164 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:11.188220 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.188264 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:11.188292 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:11.194433 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42711
I20260620 00:15:11.194473 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42711 every 8 connection(s)
I20260620 00:15:11.194834 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:15:11.195937 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.195546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:54657)
0620 00:15:11.195720 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.195792 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260620 00:15:11.196488 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43019 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43019: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
I20260620 00:15:11.196874 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.196767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:59981)
0620 00:15:11.196820 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.196851 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:15:11.197185 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.197130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:34335)
0620 00:15:11.197152 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.197172 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 00:15:11.197326 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:15:11.200176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15556
I20260620 00:15:11.952842 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:11.952505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:53111)
0620 00:15:11.952726 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:11.952773 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":146,"threads_started":1}
I20260620 00:15:12.048161 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:12.047941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:60045)
0620 00:15:12.048066 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:12.048103 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 00:15:12.197964 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:12.197726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:56811)
0620 00:15:12.197868 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:12.197905 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260620 00:15:12.953781 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:12.953515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:42647)
0620 00:15:12.953679 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:12.953731 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260620 00:15:13.049500 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:13.049121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:53131)
0620 00:15:13.049394 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:13.049449 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":74,"threads_started":1}
I20260620 00:15:13.198773 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:13.198519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:54911)
0620 00:15:13.198676 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:13.198723 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260620 00:15:13.954687 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:13.954437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:48429)
0620 00:15:13.954586 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:13.954633 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260620 00:15:14.050498 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:14.050130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:46983)
0620 00:15:14.050380 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:14.050424 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":100,"threads_started":1}
I20260620 00:15:14.199620 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:14.199360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:48225)
0620 00:15:14.199507 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:14.199545 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260620 00:15:14.955560 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:14.955302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:35687)
0620 00:15:14.955463 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:14.955510 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260620 00:15:15.051321 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:15.051069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:36867)
0620 00:15:15.051223 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:15.051271 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260620 00:15:15.200472 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:15.200207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:42693)
0620 00:15:15.200371 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:15.200417 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260620 00:15:15.956604 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:15.956296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:49095)
0620 00:15:15.956453 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:15.956496 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260620 00:15:16.052109 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:16.051852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:52349)
0620 00:15:16.052006 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:16.052059 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260620 00:15:16.201364 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:16.201115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:54397)
0620 00:15:16.201281 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:16.201316 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260620 00:15:16.957433 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:16.957189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:34401)
0620 00:15:16.957334 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:16.957383 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260620 00:15:17.053045 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:17.052781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:39057)
0620 00:15:17.052946 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:17.052995 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260620 00:15:17.202199 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:17.201888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:36025)
0620 00:15:17.202078 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:17.202126 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260620 00:15:17.958348 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:17.958099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:55227)
0620 00:15:17.958249 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:17.958297 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260620 00:15:18.053848 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:18.053589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:57913)
0620 00:15:18.053749 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:18.053797 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260620 00:15:18.203166 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:18.202901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:44685)
0620 00:15:18.203066 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:18.203114 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260620 00:15:18.959251 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:18.958994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:53177)
0620 00:15:18.959152 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:18.959202 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260620 00:15:19.054760 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:19.054498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:35547)
0620 00:15:19.054669 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:19.054707 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260620 00:15:19.204048 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:19.203758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:52183)
0620 00:15:19.203940 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:19.203998 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260620 00:15:19.960116 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:19.959845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:40725)
0620 00:15:19.960015 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:19.960065 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260620 00:15:20.055751 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:20.055415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:33765)
0620 00:15:20.055634 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:20.055697 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260620 00:15:20.205016 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:20.204775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:44055)
0620 00:15:20.204922 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:20.204967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260620 00:15:20.961100 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:20.960802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:34895)
0620 00:15:20.960988 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:20.961044 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260620 00:15:21.056905 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:21.056652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:45887)
0620 00:15:21.056798 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:21.056848 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260620 00:15:21.205971 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:21.205638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:53895)
0620 00:15:21.205869 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:21.205919 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":133,"threads_started":1}
I20260620 00:15:21.962052 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:21.961719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:52441)
0620 00:15:21.961917 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:21.962003 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":144,"threads_started":1}
I20260620 00:15:22.057719 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:22.057474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:38879)
0620 00:15:22.057626 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:22.057668 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260620 00:15:22.206897 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:22.206606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:57157)
0620 00:15:22.206797 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:22.206839 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260620 00:15:22.962899 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:22.962633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:42397)
0620 00:15:22.962802 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:22.962851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260620 00:15:23.058601 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:23.058366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:41083)
0620 00:15:23.058501 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:23.058553 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260620 00:15:23.207800 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:23.207517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:53309)
0620 00:15:23.207689 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:23.207747 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260620 00:15:23.963696 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:23.963460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:48949)
0620 00:15:23.963608 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:23.963647 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260620 00:15:24.059499 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:24.059179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:47055)
0620 00:15:24.059374 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:24.059452 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":114,"threads_started":1}
I20260620 00:15:24.208792 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:24.208483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:50483)
0620 00:15:24.208691 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:24.208740 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":123,"threads_started":1}
I20260620 00:15:24.964669 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:24.964336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:55575)
0620 00:15:24.964503 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:24.964609 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260620 00:15:25.060348 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:25.060100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:34763)
0620 00:15:25.060251 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:25.060294 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260620 00:15:25.209568 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:25.209330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:44233)
0620 00:15:25.209476 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:25.209519 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 00:15:25.965708 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:25.965349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.1:47365)
0620 00:15:25.965592 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:25.965642 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260620 00:15:26.061384 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:26.061056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.2:44055)
0620 00:15:26.061252 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:26.061300 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260620 00:15:26.210624 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:26.210387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43019 (local address 127.0.98.3:57631)
0620 00:15:26.210527 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:26.210575 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260620 00:15:26.211410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15262
I20260620 00:15:26.216029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15424
I20260620 00:15:26.220856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15556
2026-06-20T00:15:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18503 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:15:26 dist-test-slave-28bv krb5kdc[15736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:15:26 dist-test-slave-28bv krb5kdc[15736](info): set up 2 sockets
Jun 20 00:15:26 dist-test-slave-28bv krb5kdc[15736](info): commencing operation
krb5kdc: starting...
W20260620 00:15:27.268795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:15:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:15:27.399823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:27.485576 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:27.485875 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:27.485920 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:27.487324 15749 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:15:27.487407 15749 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:15:27.487432 15749 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:15:27.487450 15749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:27.487481 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:27.487514 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:27.487545 15749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:27.487576 15749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:27.489351 15749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:27.489697 15749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:27.489980 15749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:27.495620 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:27.495677 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.495674 15749 server_base.cc:1034] running on GCE node
W20260620 00:15:27.495620 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.496120 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:27.496506 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:27.497727 15749 hybrid_clock.cc:648] HybridClock initialized: now 1781914527497708 us; error 31 us; skew 500 ppm
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:27.500840 15749 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:27.501423 15749 webserver.cc:460] Webserver started at http://127.0.98.62:36905/ using document root <none> and password file <none>
I20260620 00:15:27.501667 15749 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:27.501745 15749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:27.501904 15749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:27.503077 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "2b42e7af16ef45bf9ace392979d9d3ca"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "cbe726acfcada7f590f34fe67563ffc5"
server_key_iv: "28e6889cc2a1495cf63bf3ae2db4aad7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.503463 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b42e7af16ef45bf9ace392979d9d3ca"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "cbe726acfcada7f590f34fe67563ffc5"
server_key_iv: "28e6889cc2a1495cf63bf3ae2db4aad7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.504998 15749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:15:27.505921 15764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.506237 15749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:27.506349 15749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "2b42e7af16ef45bf9ace392979d9d3ca"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "cbe726acfcada7f590f34fe67563ffc5"
server_key_iv: "28e6889cc2a1495cf63bf3ae2db4aad7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.506453 15749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:27.514489 15749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:27.516973 15749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:27.517149 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:27.521528 15749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45367
I20260620 00:15:27.521559 15816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45367 every 8 connection(s)
I20260620 00:15:27.521929 15749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:27.522862 15817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:27.525252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15749
I20260620 00:15:27.525354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:15:27.525352 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap starting.
I20260620 00:15:27.525568 392 external_mini_cluster.cc:1428] Setting key e1cd0c86d6878ddfbad965cc5f49d5ef
I20260620 00:15:27.526087 15817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:27.526502 15817 log.cc:824] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:27.527243 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: No bootstrap required, opened a new log
I20260620 00:15:27.529243 15817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:27.529421 15817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:27.529475 15817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Initialized, Role: FOLLOWER
I20260620 00:15:27.529650 15817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:27.529747 15817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:27.529796 15817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:27.529837 15817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:27.530438 15817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:27.530568 15817 leader_election.cc:302] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b42e7af16ef45bf9ace392979d9d3ca; no voters:
I20260620 00:15:27.530781 15817 leader_election.cc:288] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:15:27.530815 15822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:27.531023 15822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 LEADER]: Becoming Leader. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Running, Role: LEADER
I20260620 00:15:27.531095 15817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:27.531157 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:27.531651 15823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:15:27.531656 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b42e7af16ef45bf9ace392979d9d3ca. Latest consensus state: current_term: 1 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:15:27.531818 15823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:27.531883 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:27.532202 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:27.532785 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:27.534936 15831 catalog_manager.cc:1347] Generated new cluster ID: 559b5d16153e45f5a500f326fb41b34b
I20260620 00:15:27.534996 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:27.537986 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:27.526170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.0.98.62:45367)
0620 00:15:27.526399 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:27.526407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:27.526433 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:15:27.527039 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:27.527047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:27.527099 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:27.527255 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:27.527923 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.528910 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.529758 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.530263 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.530394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:27.533570 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:27.533575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:27.533576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:27.533590 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:27.535761 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.536226 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.536229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.536229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.536336 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.536591 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.536592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.536592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.536938 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:27.537087 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:27.537595 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 00:15:27.537752 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260620 00:15:27.554052 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:15:27.554665 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:27.568256 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Generated new TSK 0
I20260620 00:15:27.568539 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:15:27.604892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:27.690588 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:27.690824 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:27.690850 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:27.692098 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:27.692147 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:27.692231 15843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:27.693837 15843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:27.694121 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:27.694380 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:27.699538 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:27.699533 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.699666 15843 server_base.cc:1034] running on GCE node
W20260620 00:15:27.699630 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.700028 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:27.700290 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:27.701433 15843 hybrid_clock.cc:648] HybridClock initialized: now 1781914527701440 us; error 49 us; skew 500 ppm
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:27.704175 15843 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:27.704723 15843 webserver.cc:460] Webserver started at http://127.0.98.1:43907/ using document root <none> and password file <none>
I20260620 00:15:27.704994 15843 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:27.705065 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:27.705216 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:27.706291 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "03d65c6c77ee5206a0b7388a575a55ae"
server_key_iv: "2663ca6cc1b0c69ffab62ac97b055f0a"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.706661 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "03d65c6c77ee5206a0b7388a575a55ae"
server_key_iv: "2663ca6cc1b0c69ffab62ac97b055f0a"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.708007 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:15:27.708774 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.708947 15843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:27.709052 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "03d65c6c77ee5206a0b7388a575a55ae"
server_key_iv: "2663ca6cc1b0c69ffab62ac97b055f0a"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.709129 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:27.722621 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:27.725270 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:27.725455 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:27.725724 15843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:27.726069 15843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:27.726123 15843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.726159 15843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:27.726202 15843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.732582 15843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45967
I20260620 00:15:27.732646 15971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45967 every 8 connection(s)
I20260620 00:15:27.732993 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:27.740209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15843
I20260620 00:15:27.740383 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:15:27.740655 392 external_mini_cluster.cc:1428] Setting key 29fc76465dc4782c8a9d12a07d707f84
I20260620 00:15:27.743242 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:27.733795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50505 (local address 127.0.98.62:45367)
0620 00:15:27.733882 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:27.733885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:27.734015 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:15:27.734595 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:27.734596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:27.734608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:27.734669 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:27.735674 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.736203 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.737306 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.737518 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.737608 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:27.739932 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:27.739934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:27.739934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:27.739941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:27.741720 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.742209 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.742210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.742210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.742274 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.742559 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.742560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.742560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.742681 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:27.742757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:27.743052 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 00:15:27.743148 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:15:27.743460 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:27.733707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:50505)
0620 00:15:27.733865 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:27.733918 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:27.734479 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:27.734752 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:27.734756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:27.735351 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:27.735554 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:27.735559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.736311 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.736312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:27.737210 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:27.737212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.737618 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.737619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:27.737749 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:27.738278 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:27.738287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:27.739799 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:27.741836 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:27.741846 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:27.741849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:27.742112 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:27.742361 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:27.742362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:27.742362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:27.742484 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:27.742749 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:27.742756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:27.742914 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 00:15:27.743022 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:15:27.743193 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260620 00:15:27.744115 15972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:27.744227 15972 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:27.744431 15972 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:27.745064 15781 ts_manager.cc:194] Registered new tserver with Master: c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967)
I20260620 00:15:27.745708 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50505
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:15:27.767943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:27.851123 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:27.851418 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:27.851465 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:27.852720 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:27.852802 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:27.852849 15977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:27.854411 15977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:27.854700 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:27.854946 15977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:27.860728 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.860774 15977 server_base.cc:1034] running on GCE node
W20260620 00:15:27.860718 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:27.860713 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:27.861127 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:27.861435 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:27.862591 15977 hybrid_clock.cc:648] HybridClock initialized: now 1781914527862573 us; error 29 us; skew 500 ppm
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:27.865346 15977 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:27.865897 15977 webserver.cc:460] Webserver started at http://127.0.98.2:33453/ using document root <none> and password file <none>
I20260620 00:15:27.866163 15977 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:27.866236 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:27.866392 15977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:27.867513 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "11b43df81e2c470096413f9b41dfb5ee"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "7cb33f572b9637145cf8e62d5b198cde"
server_key_iv: "931756aaa6b8fe5669d0a4dc2f72b5c7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.867911 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "11b43df81e2c470096413f9b41dfb5ee"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "7cb33f572b9637145cf8e62d5b198cde"
server_key_iv: "931756aaa6b8fe5669d0a4dc2f72b5c7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.869650 15977 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 00:15:27.870628 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.870895 15977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:27.870993 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "11b43df81e2c470096413f9b41dfb5ee"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "7cb33f572b9637145cf8e62d5b198cde"
server_key_iv: "931756aaa6b8fe5669d0a4dc2f72b5c7"
server_key_version: "encryptionkey@0"
I20260620 00:15:27.871096 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:27.886963 15977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:27.889241 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:27.889421 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:27.889683 15977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:27.890019 15977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:27.890077 15977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.890123 15977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:27.890156 15977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:27.896425 15977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38091
I20260620 00:15:27.896463 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38091 every 8 connection(s)
I20260620 00:15:27.896852 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:15:27 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:27.903223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 15977
I20260620 00:15:27.903316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:15:27.903530 392 external_mini_cluster.cc:1428] Setting key 5699157d01bc1d3e76d2cc077133a6f4
I20260620 00:15:27.906901 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:27.897630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50095 (local address 127.0.98.62:45367)
0620 00:15:27.897718 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:27.897720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:27.897889 (+ 169us) server_negotiation.cc:407] Connection header received
0620 00:15:27.898338 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:27.898339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:27.898350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:27.898428 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:27.899396 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.899905 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.901008 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.901193 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.901252 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:27.903524 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:27.903526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:27.903527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:27.903535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:27.905334 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.905867 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.905868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.905869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.905933 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:27.906212 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:27.906213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:27.906213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:27.906343 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:27.906418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:27.906711 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 00:15:27.906805 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 00:15:27.907115 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:27.897555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:50095)
0620 00:15:27.897727 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:27.897777 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:27.898255 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:27.898486 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:27.898489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:27.899084 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:27.899288 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:27.899293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.900004 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.900004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:27.900929 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:27.900931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:27.901283 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:27.901284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:27.901391 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:27.901885 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:27.901893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:27.903411 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:27.905477 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:27.905489 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:27.905491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:27.905778 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:27.906014 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:27.906015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:27.906015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:27.906134 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:27.906418 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:27.906426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:27.906574 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 00:15:27.906686 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:15:27.906863 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260620 00:15:27.907735 16106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:27.907842 16106 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:27.908051 16106 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:27.908483 15781 ts_manager.cc:194] Registered new tserver with Master: 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
I20260620 00:15:27.909119 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50095
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:15:27.932152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:28.018556 16111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:28.018806 16111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:28.018836 16111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:28.020131 16111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:28.020180 16111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:28.020218 16111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:28.021879 16111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:28.022109 16111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:28.022325 16111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:28.027915 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:28.027918 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:28.027909 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:28.027937 16111 server_base.cc:1034] running on GCE node
I20260620 00:15:28.028443 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:28.028854 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:28.030004 16111 hybrid_clock.cc:648] HybridClock initialized: now 1781914528029992 us; error 24 us; skew 500 ppm
Jun 20 00:15:28 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:28.033056 16111 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:28.033619 16111 webserver.cc:460] Webserver started at http://127.0.98.3:42815/ using document root <none> and password file <none>
I20260620 00:15:28.033859 16111 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:28.033910 16111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:28.034008 16111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:15:28.035069 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "21c3d686991648a4916a25f54b662e5f"
format_stamp: "Formatted at 2026-06-20 00:15:28 on dist-test-slave-28bv"
server_key: "054adf0afea124c1cbeb24d4b7b5ab5f"
server_key_iv: "bd690b0b76f73208d1ee4b8c5f1ff27f"
server_key_version: "encryptionkey@0"
I20260620 00:15:28.035393 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21c3d686991648a4916a25f54b662e5f"
format_stamp: "Formatted at 2026-06-20 00:15:28 on dist-test-slave-28bv"
server_key: "054adf0afea124c1cbeb24d4b7b5ab5f"
server_key_iv: "bd690b0b76f73208d1ee4b8c5f1ff27f"
server_key_version: "encryptionkey@0"
I20260620 00:15:28.036900 16111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:15:28.037876 16126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:28.038292 16111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:28.038383 16111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "21c3d686991648a4916a25f54b662e5f"
format_stamp: "Formatted at 2026-06-20 00:15:28 on dist-test-slave-28bv"
server_key: "054adf0afea124c1cbeb24d4b7b5ab5f"
server_key_iv: "bd690b0b76f73208d1ee4b8c5f1ff27f"
server_key_version: "encryptionkey@0"
I20260620 00:15:28.038496 16111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:28.066362 16111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:28.069228 16111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:28.069420 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:28.069684 16111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:28.070007 16111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:28.070065 16111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:28.070108 16111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:28.070137 16111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:28.076670 16111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260620 00:15:28.076717 16239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260620 00:15:28.077101 16111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:15:28.077776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16111
I20260620 00:15:28.077885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:15:28.078110 392 external_mini_cluster.cc:1428] Setting key 2f60f520d48b0eebe1c10efe9d9f8175
Jun 20 00:15:28 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:28.087703 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.077972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48405 (local address 127.0.98.62:45367)
0620 00:15:28.078033 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.078036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.078377 (+ 341us) server_negotiation.cc:407] Connection header received
0620 00:15:28.079121 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.079122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.079135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.079176 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:28.080189 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.080791 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.081870 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.082073 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.082151 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.084588 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:28.084591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:28.084592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:28.084606 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:28.086127 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:28.086595 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:28.086596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:28.086597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:28.086663 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:28.086958 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:28.086959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:28.086959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:28.087089 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:28.087166 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.087500 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.087595 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:15:28.087953 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.077946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:48405)
0620 00:15:28.078144 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.078263 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.079008 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.079275 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.079280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.079883 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:28.080085 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.080090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.080901 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.080902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.081744 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.081747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.082171 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.082172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.082316 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.082921 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:28.082930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:28.084433 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:28.086236 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:28.086246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:28.086249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:28.086510 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:28.086749 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:28.086750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:28.086750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:28.086879 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:28.087168 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:28.087176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:28.087356 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.087479 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.087667 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260620 00:15:28.088660 16240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:28.088788 16240 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:28.088999 16240 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:28.089627 15781 ts_manager.cc:194] Registered new tserver with Master: 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563)
I20260620 00:15:28.090169 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48405
I20260620 00:15:28.090382 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:15:28.098603 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.092047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.0.98.62:45367)
0620 00:15:28.092105 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.092107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.092216 (+ 109us) server_negotiation.cc:407] Connection header received
0620 00:15:28.092284 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.092285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.092294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.092336 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:28.093014 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.093551 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.094366 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.094597 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.094668 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.095924 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:28.095925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:28.095926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:28.095932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:28.097547 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:28.097904 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:28.097905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:28.097905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:28.097956 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:28.098121 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:28.098121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:28.098122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:28.098272 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:28.098351 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.098461 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.098533 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:15:28.100648 15781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:15:28.102406 15781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:15:28.109972 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.104968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:36104)
0620 00:15:28.105747 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.105757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.105806 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.106031 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.106032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.106035 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:28.106083 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.106085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.107083 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.107085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.108555 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.108558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109640 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.109711 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109716 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.109800 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.109893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260620 00:15:28.109987 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.105154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.0.98.1:45967)
0620 00:15:28.105462 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.105468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.105866 (+ 398us) server_negotiation.cc:407] Connection header received
0620 00:15:28.105934 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.105937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.105953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.106022 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:28.106141 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.106948 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.108671 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109463 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109538 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109601 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.109835 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.109883 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":180,"threads_started":1}
I20260620 00:15:28.110018 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.105251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.0.98.3:40563)
0620 00:15:28.105463 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.105468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.105481 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:15:28.105527 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.105529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.105546 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.105649 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:28.105849 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.107145 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.108671 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109464 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109535 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109601 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.109868 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.109921 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20260620 00:15:28.110184 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.104965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.0.1:55706)
0620 00:15:28.105163 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.105209 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.105281 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.105659 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.105663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.105666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:28.105754 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.105757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.107271 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.107272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.108566 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.108568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109568 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.109684 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109701 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.109815 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.110111 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":256,"thread_start_us":56,"threads_started":1}
I20260620 00:15:28.110975 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.105262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.2:38091)
0620 00:15:28.105462 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.105468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.105521 (+ 53us) server_negotiation.cc:407] Connection header received
0620 00:15:28.105624 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.105627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.105643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.105744 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:28.105952 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.107146 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.108444 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109563 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109641 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109707 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.110828 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.110884 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260620 00:15:28.111079 16174 tablet_service.cc:1467] Processing CreateTablet for tablet 8804241ef2464b6c92f0627511d41f4a (DEFAULT_TABLE table=test-table [id=5014c76f77ab4f37adfbc8315867b54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:28.111301 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.104968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38091 (local address 127.0.0.1:35566)
0620 00:15:28.105465 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.105475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.105530 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.105824 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.105825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.105827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:28.105872 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.105873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.107294 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.107295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.108340 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.108343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.109745 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.109746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.109830 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.109834 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.109888 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.109933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"spinlock_wait_cycles":768,"thread_start_us":114,"threads_started":1}
I20260620 00:15:28.111522 16174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8804241ef2464b6c92f0627511d41f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:28.111481 15906 tablet_service.cc:1467] Processing CreateTablet for tablet 8804241ef2464b6c92f0627511d41f4a (DEFAULT_TABLE table=test-table [id=5014c76f77ab4f37adfbc8315867b54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:28.111848 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8804241ef2464b6c92f0627511d41f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:28.112171 16040 tablet_service.cc:1467] Processing CreateTablet for tablet 8804241ef2464b6c92f0627511d41f4a (DEFAULT_TABLE table=test-table [id=5014c76f77ab4f37adfbc8315867b54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:28.112519 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8804241ef2464b6c92f0627511d41f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:28.114877 16257 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap starting.
I20260620 00:15:28.115361 16258 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Bootstrap starting.
I20260620 00:15:28.115746 16257 tablet_bootstrap.cc:654] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:28.116032 16258 tablet_bootstrap.cc:654] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:28.116356 16259 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Bootstrap starting.
I20260620 00:15:28.116384 16257 log.cc:824] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:28.116642 16258 log.cc:824] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:28.117329 16257 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: No bootstrap required, opened a new log
I20260620 00:15:28.117338 16259 tablet_bootstrap.cc:654] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:28.117421 16257 ts_tablet_manager.cc:1397] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 00:15:28.117497 16258 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: No bootstrap required, opened a new log
I20260620 00:15:28.117573 16258 ts_tablet_manager.cc:1397] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:15:28.118146 16259 log.cc:824] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:28.118959 16259 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: No bootstrap required, opened a new log
I20260620 00:15:28.119048 16259 ts_tablet_manager.cc:1397] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:15:28.119509 16258 raft_consensus.cc:348] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.119509 16257 raft_consensus.cc:348] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.119712 16258 raft_consensus.cc:374] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:28.119712 16257 raft_consensus.cc:374] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:28.119752 16258 raft_consensus.cc:729] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3d686991648a4916a25f54b662e5f, State: Initialized, Role: FOLLOWER
I20260620 00:15:28.119752 16257 raft_consensus.cc:729] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a692c1f7a42ed9eb73dd3dda335ae, State: Initialized, Role: FOLLOWER
I20260620 00:15:28.119881 16257 consensus_queue.cc:260] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.119907 16258 consensus_queue.cc:260] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.120196 16258 ts_tablet_manager.cc:1428] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:15:28.120230 15972 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260620 00:15:28.120314 16240 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260620 00:15:28.120596 16257 ts_tablet_manager.cc:1428] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:15:28.121120 16259 raft_consensus.cc:348] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.121320 16259 raft_consensus.cc:374] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:28.121368 16259 raft_consensus.cc:729] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b43df81e2c470096413f9b41dfb5ee, State: Initialized, Role: FOLLOWER
I20260620 00:15:28.121521 16259 consensus_queue.cc:260] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.121831 16259 ts_tablet_manager.cc:1428] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:15:28.121901 16106 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
W20260620 00:15:28.147435 16107 tablet.cc:2369] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:15:28.233816 15973 tablet.cc:2369] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:15:28.327768 16241 tablet.cc:2369] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:28.384967 16263 raft_consensus.cc:482] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:28.385104 16263 raft_consensus.cc:504] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.385494 16263 leader_election.cc:288] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091), 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563)
I20260620 00:15:28.388733 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.385791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41949 (local address 127.0.98.2:38091)
0620 00:15:28.385844 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.385846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.385956 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:15:28.386071 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.386072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.386091 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.386170 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:28.386378 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.386853 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.387931 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.388305 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.388398 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.388408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.388607 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.388649 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:15:28.388753 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.385857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52909 (local address 127.0.98.3:40563)
0620 00:15:28.385910 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.385912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.385985 (+ 73us) server_negotiation.cc:407] Connection header received
0620 00:15:28.386075 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.386076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.386086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.386132 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:28.386344 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.386830 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.387902 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.388300 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.388357 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.388392 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.388637 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.388680 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:15:28.388810 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.385751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.98.1:52909)
0620 00:15:28.385890 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.385905 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.385976 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.386155 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.386157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.386159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:28.386246 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.386249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.386917 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.386918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.387800 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.387803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.388389 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.388390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.388491 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.388496 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.388629 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.388671 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260620 00:15:28.388820 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.385688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38091 (local address 127.0.98.1:41949)
0620 00:15:28.385882 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.385895 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:28.385981 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:28.386189 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:28.386190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:28.386192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:28.386278 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.386280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.386979 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.386979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.387811 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:28.387813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.388423 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.388424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:28.388497 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.388501 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:28.388589 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:15:28.388639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":23,"thread_start_us":116,"threads_started":1}
I20260620 00:15:28.389202 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8804241ef2464b6c92f0627511d41f4a" candidate_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3d686991648a4916a25f54b662e5f" is_pre_election: true
I20260620 00:15:28.389200 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8804241ef2464b6c92f0627511d41f4a" candidate_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b43df81e2c470096413f9b41dfb5ee" is_pre_election: true
I20260620 00:15:28.389393 16060 raft_consensus.cc:2454] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42a692c1f7a42ed9eb73dd3dda335ae in term 0.
I20260620 00:15:28.389401 16194 raft_consensus.cc:2454] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42a692c1f7a42ed9eb73dd3dda335ae in term 0.
I20260620 00:15:28.389639 15860 leader_election.cc:302] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b43df81e2c470096413f9b41dfb5ee, c42a692c1f7a42ed9eb73dd3dda335ae; no voters:
I20260620 00:15:28.389789 16263 raft_consensus.cc:2781] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:28.389851 16263 raft_consensus.cc:482] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:28.389911 16263 raft_consensus.cc:3037] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:28.390708 16263 raft_consensus.cc:504] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.390878 16263 leader_election.cc:288] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 1 election: Requested vote from peers 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091), 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563)
I20260620 00:15:28.391031 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8804241ef2464b6c92f0627511d41f4a" candidate_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b43df81e2c470096413f9b41dfb5ee"
I20260620 00:15:28.391105 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8804241ef2464b6c92f0627511d41f4a" candidate_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c3d686991648a4916a25f54b662e5f"
I20260620 00:15:28.391134 16060 raft_consensus.cc:3037] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:28.391217 16194 raft_consensus.cc:3037] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:28.392053 16194 raft_consensus.cc:2454] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42a692c1f7a42ed9eb73dd3dda335ae in term 1.
I20260620 00:15:28.392055 16060 raft_consensus.cc:2454] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42a692c1f7a42ed9eb73dd3dda335ae in term 1.
I20260620 00:15:28.392246 15860 leader_election.cc:302] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b43df81e2c470096413f9b41dfb5ee, c42a692c1f7a42ed9eb73dd3dda335ae; no voters:
I20260620 00:15:28.392387 16263 raft_consensus.cc:2781] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:28.392593 16263 raft_consensus.cc:686] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 LEADER]: Becoming Leader. State: Replica: c42a692c1f7a42ed9eb73dd3dda335ae, State: Running, Role: LEADER
I20260620 00:15:28.392719 16263 consensus_queue.cc:237] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:28.393682 15780 catalog_manager.cc:5495] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae reported cstate change: term changed from 0 to 1, leader changed from <none> to c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } health_report { overall_health: UNKNOWN } } }
I20260620 00:15:28.464915 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.462342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.0.98.1:45967)
0620 00:15:28.462438 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:28.462440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:28.462449 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:15:28.462507 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:28.462508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:28.462529 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:28.462593 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:28.462758 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.463230 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.464140 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:28.464332 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:28.464397 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:28.464579 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:28.464686 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:28.464748 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:28.464779 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:15:28.464798 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 00:15:28.467026 16194 raft_consensus.cc:1261] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Refusing update from remote peer c42a692c1f7a42ed9eb73dd3dda335ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:28.467026 16060 raft_consensus.cc:1261] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Refusing update from remote peer c42a692c1f7a42ed9eb73dd3dda335ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:28.467437 16268 consensus_queue.cc:1035] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:28.467608 16263 consensus_queue.cc:1035] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:28.470011 16271 mvcc.cc:205] Tried to move back new op lower bound from 7298721908597460992 to 7298721908297117696. Current Snapshot: MvccSnapshot[applied={T|T < 7298721908597460992}]
I20260620 00:15:28.470148 16272 mvcc.cc:205] Tried to move back new op lower bound from 7298721908597460992 to 7298721908297117696. Current Snapshot: MvccSnapshot[applied={T|T < 7298721908597460992}]
I20260620 00:15:28.470466 16273 mvcc.cc:205] Tried to move back new op lower bound from 7298721908597460992 to 7298721908297117696. Current Snapshot: MvccSnapshot[applied={T|T < 7298721908597460992}]
I20260620 00:15:28.472584 15781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:28.472703 15781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:28.473549 15781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Sending DeleteTablet for 3 replicas of tablet 8804241ef2464b6c92f0627511d41f4a
I20260620 00:15:28.473922 16174 tablet_service.cc:1514] Processing DeleteTablet for tablet 8804241ef2464b6c92f0627511d41f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55706
I20260620 00:15:28.474025 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8804241ef2464b6c92f0627511d41f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35566
I20260620 00:15:28.474045 15906 tablet_service.cc:1514] Processing DeleteTablet for tablet 8804241ef2464b6c92f0627511d41f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36104
I20260620 00:15:28.474244 16282 tablet_replica.cc:330] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: stopping tablet replica
I20260620 00:15:28.474341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15843
I20260620 00:15:28.474370 16282 raft_consensus.cc:2229] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:28.474529 16281 tablet_replica.cc:330] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: stopping tablet replica
I20260620 00:15:28.475136 16282 raft_consensus.cc:2258] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:28.475416 16281 raft_consensus.cc:2229] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:28.475596 16281 raft_consensus.cc:2258] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:28.475739 16282 ts_tablet_manager.cc:1905] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:28.476729 16281 ts_tablet_manager.cc:1905] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:28.477433 16282 ts_tablet_manager.cc:1918] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:28.477523 16282 log.cc:1196] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/8804241ef2464b6c92f0627511d41f4a
I20260620 00:15:28.477821 16282 ts_tablet_manager.cc:1939] T 8804241ef2464b6c92f0627511d41f4a P 11b43df81e2c470096413f9b41dfb5ee: Deleting consensus metadata
I20260620 00:15:28.478400 15766 catalog_manager.cc:4841] TS 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): tablet 8804241ef2464b6c92f0627511d41f4a (table test-table [id=5014c76f77ab4f37adfbc8315867b54f]) successfully deleted
I20260620 00:15:28.478436 16281 ts_tablet_manager.cc:1918] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:28.478519 16281 log.cc:1196] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/8804241ef2464b6c92f0627511d41f4a
I20260620 00:15:28.478776 16281 ts_tablet_manager.cc:1939] T 8804241ef2464b6c92f0627511d41f4a P 21c3d686991648a4916a25f54b662e5f: Deleting consensus metadata
I20260620 00:15:28.479434 15767 catalog_manager.cc:4841] TS 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): tablet 8804241ef2464b6c92f0627511d41f4a (table test-table [id=5014c76f77ab4f37adfbc8315867b54f]) successfully deleted
W20260620 00:15:28.480298 15765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45967 (error 108)
I20260620 00:15:28.480521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15977
I20260620 00:15:28.481384 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:28.481229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:38970)
0620 00:15:28.481293 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:28.481336 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 00:15:28.481531 15765 catalog_manager.cc:4584] TS c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8804241ef2464b6c92f0627511d41f4a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection refused (error 111)
I20260620 00:15:28.485770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16111
I20260620 00:15:28.491089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 15749
I20260620 00:15:48.497295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--webserver_interface=127.0.98.62
--webserver_port=36905
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:15:48.581799 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:48.582062 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:48.582086 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:48.583389 16285 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:15:48.583446 16285 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:15:48.583460 16285 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:15:48.583473 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:15:48.583487 16285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:48.583499 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:48.583559 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:15:48.583577 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:15:48.585281 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:48.585583 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:48.585894 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:48.596043 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:48.596020 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:48.596347 16285 server_base.cc:1034] running on GCE node
W20260620 00:15:48.596020 16294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:48.596755 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:48.597150 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:48.598352 16285 hybrid_clock.cc:648] HybridClock initialized: now 1781914548598319 us; error 55 us; skew 500 ppm
Jun 20 00:15:48 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:48.605676 16285 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:15:48.606225 16285 webserver.cc:460] Webserver started at http://127.0.98.62:36905/ using document root <none> and password file <none>
I20260620 00:15:48.606508 16285 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:48.606606 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:48.608619 16285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:15:48.609499 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:48.609753 16285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:48.609922 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "2b42e7af16ef45bf9ace392979d9d3ca"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "cbe726acfcada7f590f34fe67563ffc5"
server_key_iv: "28e6889cc2a1495cf63bf3ae2db4aad7"
server_key_version: "encryptionkey@0"
I20260620 00:15:48.610402 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:48.622968 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:48.625919 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:48.626259 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:48.630966 16285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45367
I20260620 00:15:48.631045 16353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45367 every 8 connection(s)
I20260620 00:15:48.631497 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:15:48.633479 16354 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:15:48.633612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16285
I20260620 00:15:48.633951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45967
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=43907
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:48.634428 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap starting.
I20260620 00:15:48.644012 16354 log.cc:824] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:48.646152 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:48.646479 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap complete.
I20260620 00:15:48.649055 16354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:48.649326 16354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Initialized, Role: FOLLOWER
I20260620 00:15:48.649537 16354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:48.649639 16354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:15:48.649700 16354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:15:48.649752 16354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:15:48.650898 16354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:48.651032 16354 leader_election.cc:302] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b42e7af16ef45bf9ace392979d9d3ca; no voters:
I20260620 00:15:48.651265 16354 leader_election.cc:288] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:15:48.651306 16358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:15:48.651428 16358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 LEADER]: Becoming Leader. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Running, Role: LEADER
I20260620 00:15:48.651557 16354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:15:48.651520 16358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:15:48.651962 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:15:48.652064 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:48.652303 16360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b42e7af16ef45bf9ace392979d9d3ca. Latest consensus state: current_term: 2 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:15:48.652421 16360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:15:48.652577 16365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:15:48.653441 16365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:15:48.653563 16365 catalog_manager.cc:1259] Loaded cluster ID: 559b5d16153e45f5a500f326fb41b34b
I20260620 00:15:48.653589 16365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:15:48.654752 16365 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:15:48.654968 16365 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Loaded TSK: 0
I20260620 00:15:48.665190 16365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Generated new TSK 1
I20260620 00:15:48.665335 16365 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:15:48.730573 16356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:48.730861 16356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:48.730917 16356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:48.732365 16356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:48.732451 16356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:48.732506 16356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:15:48.734261 16356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43907
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:48.734576 16356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:48.734839 16356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:48.741487 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:48.741487 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:48.741559 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:48.742427 16356 server_base.cc:1034] running on GCE node
I20260620 00:15:48.742782 16356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:48.743155 16356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:48.744360 16356 hybrid_clock.cc:648] HybridClock initialized: now 1781914548744331 us; error 46 us; skew 500 ppm
Jun 20 00:15:48 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:48.747685 16356 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:15:48.748250 16356 webserver.cc:460] Webserver started at http://127.0.98.1:43907/ using document root <none> and password file <none>
I20260620 00:15:48.748521 16356 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:48.748639 16356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:48.750147 16356 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:48.750917 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:48.751103 16356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:48.751210 16356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "03d65c6c77ee5206a0b7388a575a55ae"
server_key_iv: "2663ca6cc1b0c69ffab62ac97b055f0a"
server_key_version: "encryptionkey@0"
I20260620 00:15:48.751655 16356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:48.765268 16356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:48.767999 16356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:48.768208 16356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:48.768510 16356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:48.769191 16398 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:15:48.770946 16356 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:15:48.771024 16356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:15:48.771057 16356 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:15:48.771849 16356 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:15:48.771903 16356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:15:48.771951 16398 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap starting.
I20260620 00:15:48.780946 16356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45967
I20260620 00:15:48.781018 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45967 every 8 connection(s)
I20260620 00:15:48.781581 16356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:15:48.785432 16398 log.cc:824] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:48.788383 16398 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:15:48.788810 16398 tablet_bootstrap.cc:492] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap complete.
I20260620 00:15:48.790644 16398 ts_tablet_manager.cc:1397] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.012s
I20260620 00:15:48.790764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16356
I20260620 00:15:48.791133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38091
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=33453
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:15:48 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:48.794770 16398 raft_consensus.cc:348] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:48.795235 16398 raft_consensus.cc:729] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a692c1f7a42ed9eb73dd3dda335ae, State: Initialized, Role: FOLLOWER
I20260620 00:15:48.796128 16398 consensus_queue.cc:260] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260620 00:15:48.796723 16398 ts_tablet_manager.cc:1428] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20260620 00:15:48.797991 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.782345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:43147)
0620 00:15:48.782534 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:48.782594 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:48.783445 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:48.784033 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:48.784039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:48.784622 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:48.784972 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.784981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.786732 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.786733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.788118 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.788121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.789175 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.789177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.789358 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.790073 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:48.790085 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:48.792505 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:48.795463 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:48.795472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:48.795475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:48.795945 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:48.796381 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:48.796383 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:48.796383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:48.796601 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:48.797227 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:48.797235 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:48.797432 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 00:15:48.797559 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 00:15:48.797754 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260620 00:15:48.798166 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.782531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43147 (local address 127.0.98.62:45367)
0620 00:15:48.782804 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:48.782814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:48.782859 (+ 45us) server_negotiation.cc:407] Connection header received
0620 00:15:48.783727 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:48.783734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:48.783796 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:48.783945 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:48.785139 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.786564 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.788268 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.788978 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.789103 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.792726 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:48.792733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:48.792734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:48.792752 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:48.795283 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:48.796099 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:48.796103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:48.796104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:48.796273 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:48.796704 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:48.796705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:48.796705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:48.797139 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:48.797257 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:48.797693 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 00:15:48.797893 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
I20260620 00:15:48.798952 16506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:48.799103 16506 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:48.799360 16506 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:48.800384 16318 ts_manager.cc:194] Registered new tserver with Master: c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967)
I20260620 00:15:48.801702 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43147
I20260620 00:15:48.803136 16506 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260620 00:15:48.808137 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.801625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.0.98.1:45967)
0620 00:15:48.803812 (+ 2187us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:48.803817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:48.803827 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:15:48.803867 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:48.803870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:48.803886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:48.803996 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:48.804248 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.805016 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.807021 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.807657 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.807728 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.807773 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:48.808011 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 00:15:48.808073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2124,"thread_start_us":98,"threads_started":1}
I20260620 00:15:48.808167 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.801258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:44578)
0620 00:15:48.803529 (+ 2271us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:48.803586 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:48.803659 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:48.804039 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:48.804044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:48.804048 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:48.804147 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.804150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.805163 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.805165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.806892 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.806897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.807761 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.807763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.807873 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.807890 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 00:15:48.807978 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:15:48.808060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"thread_start_us":132,"threads_started":1}
I20260620 00:15:48.808877 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet 8804241ef2464b6c92f0627511d41f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44578
I20260620 00:15:48.809532 16515 tablet_replica.cc:330] stopping tablet replica
I20260620 00:15:48.809650 16515 raft_consensus.cc:2229] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:48.809724 16515 raft_consensus.cc:2258] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:48.810052 16515 ts_tablet_manager.cc:1905] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:48.812616 16515 ts_tablet_manager.cc:1918] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:48.812737 16515 log.cc:1196] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal/wals/8804241ef2464b6c92f0627511d41f4a
I20260620 00:15:48.812994 16515 ts_tablet_manager.cc:1939] T 8804241ef2464b6c92f0627511d41f4a P c42a692c1f7a42ed9eb73dd3dda335ae: Deleting consensus metadata
I20260620 00:15:48.813638 16302 catalog_manager.cc:4841] TS c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967): tablet 8804241ef2464b6c92f0627511d41f4a (table test-table [id=5014c76f77ab4f37adfbc8315867b54f]) successfully deleted
W20260620 00:15:48.885813 16511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:48.886092 16511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:48.886134 16511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:48.887370 16511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:48.887449 16511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:48.887497 16511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:15:48.889160 16511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33453
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:48.889449 16511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:48.889696 16511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:48.894948 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:48.895066 16511 server_base.cc:1034] running on GCE node
W20260620 00:15:48.894939 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:48.894939 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:48.895511 16511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:48.895768 16511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:48.896909 16511 hybrid_clock.cc:648] HybridClock initialized: now 1781914548896895 us; error 31 us; skew 500 ppm
Jun 20 00:15:48 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:48.899538 16511 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:15:48.900080 16511 webserver.cc:460] Webserver started at http://127.0.98.2:33453/ using document root <none> and password file <none>
I20260620 00:15:48.900337 16511 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:48.900419 16511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:48.902034 16511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:48.902725 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:48.902916 16511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:48.903061 16511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "11b43df81e2c470096413f9b41dfb5ee"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "7cb33f572b9637145cf8e62d5b198cde"
server_key_iv: "931756aaa6b8fe5669d0a4dc2f72b5c7"
server_key_version: "encryptionkey@0"
I20260620 00:15:48.903438 16511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:48.913308 16511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:48.915781 16511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:48.915966 16511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:48.916227 16511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:48.916561 16511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:48.916620 16511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:48.916661 16511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:48.916703 16511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:48.922850 16511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38091
I20260620 00:15:48.922931 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38091 every 8 connection(s)
I20260620 00:15:48.923319 16511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:15:48.929425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16511
Jun 20 00:15:48 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:48.929821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40563
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=42815
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:15:48.933991 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.924199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46731 (local address 127.0.98.62:45367)
0620 00:15:48.924249 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:48.924251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:48.924451 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:15:48.924952 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:48.924953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:48.924965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:48.925076 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:48.926076 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.926566 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.927602 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.927820 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.927880 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.930526 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:48.930528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:48.930528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:48.930536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:48.932377 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:48.932881 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:48.932882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:48.932883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:48.932948 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:48.933203 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:48.933204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:48.933204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:48.933365 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:48.933431 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:48.933722 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 00:15:48.933833 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:15:48.934039 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:48.924139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:46731)
0620 00:15:48.924316 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:48.924360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:48.924865 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:48.925158 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:48.925162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:48.925755 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:48.925954 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.925959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.926649 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.926650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.927513 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:48.927515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:48.927899 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:48.927900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:48.928024 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:48.928554 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:48.928564 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:48.930398 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:48.932474 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:48.932483 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:48.932485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:48.932804 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:48.933021 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:48.933021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:48.933022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:48.933129 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:48.933427 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:48.933431 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:48.933572 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 00:15:48.933652 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:15:48.933857 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260620 00:15:48.934633 16644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:48.934729 16644 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:48.934945 16644 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:48.935411 16318 ts_manager.cc:194] Registered new tserver with Master: 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
I20260620 00:15:48.935804 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46731
W20260620 00:15:49.017642 16647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:15:49.017930 16647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:15:49.017973 16647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:15:49.019297 16647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:15:49.019378 16647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:15:49.019424 16647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:15:49.021243 16647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42815
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:15:49.021585 16647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:15:49.021847 16647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:15:49.028120 16653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:15:49.028230 16652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:49.028349 16647 server_base.cc:1034] running on GCE node
W20260620 00:15:49.028146 16655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:15:49.028729 16647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:15:49.029028 16647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:15:49.030227 16647 hybrid_clock.cc:648] HybridClock initialized: now 1781914549030194 us; error 64 us; skew 500 ppm
Jun 20 00:15:49 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:15:49.033524 16647 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:15:49.034142 16647 webserver.cc:460] Webserver started at http://127.0.98.3:42815/ using document root <none> and password file <none>
I20260620 00:15:49.034410 16647 fs_manager.cc:359] Metadata directory not provided
I20260620 00:15:49.034507 16647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:15:49.036000 16647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:15:49.036796 16662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:49.037091 16647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:15:49.037227 16647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "21c3d686991648a4916a25f54b662e5f"
format_stamp: "Formatted at 2026-06-20 00:15:28 on dist-test-slave-28bv"
server_key: "054adf0afea124c1cbeb24d4b7b5ab5f"
server_key_iv: "bd690b0b76f73208d1ee4b8c5f1ff27f"
server_key_version: "encryptionkey@0"
I20260620 00:15:49.037650 16647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:15:49.048256 16647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:15:49.050580 16647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:15:49.050773 16647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:15:49.051023 16647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:15:49.051417 16647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:15:49.051481 16647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:49.051556 16647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:15:49.051585 16647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:15:49.058480 16647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260620 00:15:49.058534 16775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260620 00:15:49.058899 16647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:15:49 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:15:49.066078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16647
I20260620 00:15:49.069691 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.059712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60105 (local address 127.0.98.62:45367)
0620 00:15:49.059787 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.059789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.060084 (+ 295us) server_negotiation.cc:407] Connection header received
0620 00:15:49.060729 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.060731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.060743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.060804 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:49.061835 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.062341 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.063409 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.063626 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.063708 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.066145 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:49.066152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:49.066152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:49.066160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:49.068045 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.068499 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.068499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.068500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.068612 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.068894 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.068895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.068895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.069031 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:49.069083 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.069497 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.069595 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:15:49.069769 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.059721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:60105)
0620 00:15:49.059871 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.059910 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:49.060605 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:49.060881 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:49.060887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:49.061509 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:15:49.061721 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.061726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.062454 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.062455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.063304 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.063307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.063719 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.063719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.063853 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.064435 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:15:49.064444 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:15:49.066009 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:15:49.068133 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:49.068142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:49.068144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:49.068421 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:49.068673 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:15:49.068674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:15:49.068675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:15:49.068825 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:15:49.069094 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:15:49.069100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:15:49.069270 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 00:15:49.069395 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 00:15:49.069560 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260620 00:15:49.070267 16776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260620 00:15:49.070436 16776 heartbeater.cc:464] Registering TS with master...
I20260620 00:15:49.070657 16776 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260620 00:15:49.071139 16318 ts_manager.cc:194] Registered new tserver with Master: 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563)
I20260620 00:15:49.071573 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60105
I20260620 00:15:49.074218 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.066529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.0.98.62:45367)
0620 00:15:49.066658 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.066660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.066752 (+ 92us) server_negotiation.cc:407] Connection header received
0620 00:15:49.066800 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.066801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.066815 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.066894 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:49.067646 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.068365 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.069468 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.069774 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.069855 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.071415 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:49.071417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:49.071417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:49.071424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:49.073093 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.073480 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.073481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.073481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.073555 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.073734 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.073735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.073735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.073841 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:49.073969 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.074018 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.074113 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 00:15:49.075042 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:15:49.078114 16246 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45367)
I20260620 00:15:49.078142 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.075412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.0.98.62:45367)
0620 00:15:49.075473 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.075475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.075622 (+ 147us) server_negotiation.cc:407] Connection header received
0620 00:15:49.075659 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.075660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.075691 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.075764 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:49.075952 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.076450 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.077286 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.077498 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.077563 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.077703 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:49.077752 (+ 49us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 00:15:49.078055 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41358: token expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 00:15:49.078222 16780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41358: token expired
I20260620 00:15:49.085017 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.078401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.98.62:45367)
0620 00:15:49.078458 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.078459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.078490 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:15:49.078550 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.078551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.078561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.078619 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:15:49.079156 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.079615 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.080469 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.080731 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.080804 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.082102 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:15:49.082104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:15:49.082104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:15:49.082112 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:15:49.083652 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.084049 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.084050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.084050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.084114 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:15:49.084343 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:15:49.084344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:15:49.084345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:15:49.084514 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:15:49.084682 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.084796 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.084896 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:15:49.086282 16318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:15:49.087930 16318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:15:49.090533 16440 tablet_service.cc:1467] Processing CreateTablet for tablet c220d5b278dc46d38b14218abac51a1c (DEFAULT_TABLE table=test-table [id=e09c27d7df7f4a61b6ef058b56dc789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:49.090794 16440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c220d5b278dc46d38b14218abac51a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:49.092571 16398 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap starting.
I20260620 00:15:49.092943 16398 tablet_bootstrap.cc:654] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:49.093554 16398 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: No bootstrap required, opened a new log
I20260620 00:15:49.093601 16398 ts_tablet_manager.cc:1397] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:15:49.093713 16398 raft_consensus.cc:348] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.093796 16398 raft_consensus.cc:374] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:49.093853 16398 raft_consensus.cc:729] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a692c1f7a42ed9eb73dd3dda335ae, State: Initialized, Role: FOLLOWER
I20260620 00:15:49.093906 16398 consensus_queue.cc:260] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.094022 16398 ts_tablet_manager.cc:1428] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:15:49.094875 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.090182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.0.1:58970)
0620 00:15:49.090722 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.090733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:49.090792 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:49.091107 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:49.091108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:49.091110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:49.091146 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.091147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.092177 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.092178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.093491 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.093494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.094571 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.094572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.094646 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.094650 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:49.094735 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 00:15:49.094792 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":437,"thread_start_us":103,"threads_started":1}
I20260620 00:15:49.095114 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.090363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.3:40563)
0620 00:15:49.090583 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.090589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.090853 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:15:49.090899 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.090903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.090919 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.091002 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:49.091204 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.092083 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.093591 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.094456 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.094524 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.094567 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.094797 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.094844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260620 00:15:49.095784 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.090182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38091 (local address 127.0.0.1:37122)
0620 00:15:49.090223 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.090236 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:49.090274 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:49.090860 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:49.090861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:49.090863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:49.090922 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.090924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.092238 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.092238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.093491 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.093494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.095518 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.095520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.095604 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.095608 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:15:49.095667 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:15:49.095719 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":25}
I20260620 00:15:49.095988 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.090331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.2:38091)
0620 00:15:49.090679 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.090684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.090694 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:15:49.090738 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.090741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.090758 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.090851 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:49.090994 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.092117 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.094628 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.095387 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.095467 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.095541 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.095763 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.095814 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":147,"threads_started":1}
I20260620 00:15:49.095992 16710 tablet_service.cc:1467] Processing CreateTablet for tablet c220d5b278dc46d38b14218abac51a1c (DEFAULT_TABLE table=test-table [id=e09c27d7df7f4a61b6ef058b56dc789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:49.096391 16710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c220d5b278dc46d38b14218abac51a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:49.096764 16578 tablet_service.cc:1467] Processing CreateTablet for tablet c220d5b278dc46d38b14218abac51a1c (DEFAULT_TABLE table=test-table [id=e09c27d7df7f4a61b6ef058b56dc789a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:15:49.097122 16578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c220d5b278dc46d38b14218abac51a1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:15:49.100476 16786 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Bootstrap starting.
I20260620 00:15:49.100852 16787 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Bootstrap starting.
I20260620 00:15:49.101347 16786 tablet_bootstrap.cc:654] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:49.101583 16787 tablet_bootstrap.cc:654] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Neither blocks nor log segments found. Creating new log.
I20260620 00:15:49.101922 16786 log.cc:824] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:49.102145 16787 log.cc:824] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:15:49.102830 16786 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: No bootstrap required, opened a new log
I20260620 00:15:49.102926 16787 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: No bootstrap required, opened a new log
I20260620 00:15:49.102919 16786 ts_tablet_manager.cc:1397] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:15:49.102990 16787 ts_tablet_manager.cc:1397] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:15:49.105103 16786 raft_consensus.cc:348] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.105103 16787 raft_consensus.cc:348] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.105336 16787 raft_consensus.cc:374] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:49.105336 16786 raft_consensus.cc:374] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:15:49.105377 16786 raft_consensus.cc:729] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c3d686991648a4916a25f54b662e5f, State: Initialized, Role: FOLLOWER
I20260620 00:15:49.105377 16787 raft_consensus.cc:729] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b43df81e2c470096413f9b41dfb5ee, State: Initialized, Role: FOLLOWER
I20260620 00:15:49.105511 16786 consensus_queue.cc:260] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.105540 16787 consensus_queue.cc:260] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.105861 16786 ts_tablet_manager.cc:1428] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:15:49.105868 16787 ts_tablet_manager.cc:1428] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:15:49.106015 16644 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260620 00:15:49.106034 16776 heartbeater.cc:503] Master 127.0.98.62:45367 was elected leader, sending a full tablet report...
I20260620 00:15:49.154951 16790 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:15:49.155141 16790 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.155596 16790 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091), c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967)
I20260620 00:15:49.159902 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.155812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38091 (local address 127.0.98.3:37759)
0620 00:15:49.155911 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.155924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:49.156021 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:49.156181 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:49.156182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:49.156183 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:49.156258 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.156262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.157239 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.157240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.158564 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.158567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.159647 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.159648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.159728 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.159731 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:15:49.159802 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:15:49.159838 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":5}
I20260620 00:15:49.159931 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.155998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33027 (local address 127.0.98.1:45967)
0620 00:15:49.156047 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.156050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.156144 (+ 94us) server_negotiation.cc:407] Connection header received
0620 00:15:49.156218 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.156219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.156230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.156347 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:49.156575 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.157260 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.158706 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.159309 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.159439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.159452 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.159767 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.159862 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:15:49.159987 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.155847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.98.3:33027)
0620 00:15:49.156059 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.156073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:15:49.156161 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:15:49.156400 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:15:49.156401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:15:49.156402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:15:49.156471 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.156473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.157352 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.157353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.158565 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:15:49.158567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.159548 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.159549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:15:49.159652 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.159657 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:15:49.159726 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:15:49.159778 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260620 00:15:49.159965 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.155861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37759 (local address 127.0.98.2:38091)
0620 00:15:49.155942 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.155944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.156006 (+ 62us) server_negotiation.cc:407] Connection header received
0620 00:15:49.156096 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.156098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.156108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.156160 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:15:49.156346 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.157116 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.158706 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.159360 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.159439 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.159453 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.159847 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.159897 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:15:49.160354 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c220d5b278dc46d38b14218abac51a1c" candidate_uuid: "21c3d686991648a4916a25f54b662e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" is_pre_election: true
I20260620 00:15:49.160600 16460 raft_consensus.cc:2454] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3d686991648a4916a25f54b662e5f in term 0.
I20260620 00:15:49.160702 16598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c220d5b278dc46d38b14218abac51a1c" candidate_uuid: "21c3d686991648a4916a25f54b662e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b43df81e2c470096413f9b41dfb5ee" is_pre_election: true
I20260620 00:15:49.160902 16598 raft_consensus.cc:2454] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21c3d686991648a4916a25f54b662e5f in term 0.
I20260620 00:15:49.160938 16663 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c3d686991648a4916a25f54b662e5f, c42a692c1f7a42ed9eb73dd3dda335ae; no voters:
I20260620 00:15:49.161193 16790 raft_consensus.cc:2781] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:15:49.161266 16790 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:15:49.161296 16790 raft_consensus.cc:3037] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:49.162034 16790 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.162202 16790 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [CANDIDATE]: Term 1 election: Requested vote from peers 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091), c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967)
I20260620 00:15:49.162357 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c220d5b278dc46d38b14218abac51a1c" candidate_uuid: "21c3d686991648a4916a25f54b662e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
I20260620 00:15:49.162411 16598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c220d5b278dc46d38b14218abac51a1c" candidate_uuid: "21c3d686991648a4916a25f54b662e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11b43df81e2c470096413f9b41dfb5ee"
I20260620 00:15:49.162470 16460 raft_consensus.cc:3037] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:49.162523 16598 raft_consensus.cc:3037] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:15:49.163321 16598 raft_consensus.cc:2454] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c3d686991648a4916a25f54b662e5f in term 1.
I20260620 00:15:49.163516 16664 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f; no voters:
I20260620 00:15:49.163638 16460 raft_consensus.cc:2454] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21c3d686991648a4916a25f54b662e5f in term 1.
I20260620 00:15:49.163743 16790 raft_consensus.cc:2781] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:15:49.164062 16790 raft_consensus.cc:686] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 1 LEADER]: Becoming Leader. State: Replica: 21c3d686991648a4916a25f54b662e5f, State: Running, Role: LEADER
I20260620 00:15:49.164275 16790 consensus_queue.cc:237] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:15:49.165309 16318 catalog_manager.cc:5495] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 21c3d686991648a4916a25f54b662e5f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "21c3d686991648a4916a25f54b662e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } health_report { overall_health: UNKNOWN } } }
I20260620 00:15:49.172957 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:15:49.170341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.3:40563)
0620 00:15:49.170409 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:15:49.170411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:15:49.170421 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:15:49.170467 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:15:49.170468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:15:49.170487 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:15:49.170570 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:15:49.170708 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.171252 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.172044 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:15:49.172244 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:15:49.172323 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:15:49.172510 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:15:49.172678 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:15:49.172780 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:15:49.172800 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 00:15:49.172832 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 00:15:49.173974 16645 tablet.cc:2369] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:15:49.175292 16598 raft_consensus.cc:1261] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Refusing update from remote peer 21c3d686991648a4916a25f54b662e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:49.175292 16460 raft_consensus.cc:1261] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Refusing update from remote peer 21c3d686991648a4916a25f54b662e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:15:49.175742 16793 consensus_queue.cc:1035] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:49.176091 16790 consensus_queue.cc:1035] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:15:49.178659 16796 mvcc.cc:205] Tried to move back new op lower bound from 7298721993418485760 to 7298721993377763328. Current Snapshot: MvccSnapshot[applied={T|T < 7298721993418485760}]
I20260620 00:15:49.179646 16798 mvcc.cc:205] Tried to move back new op lower bound from 7298721993418485760 to 7298721993377763328. Current Snapshot: MvccSnapshot[applied={T|T < 7298721993418485760}]
I20260620 00:15:49.181599 16317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:49.181732 16317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:15:49.182713 16317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Sending DeleteTablet for 3 replicas of tablet c220d5b278dc46d38b14218abac51a1c
I20260620 00:15:49.183072 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet c220d5b278dc46d38b14218abac51a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44578
I20260620 00:15:49.183108 16578 tablet_service.cc:1514] Processing DeleteTablet for tablet c220d5b278dc46d38b14218abac51a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37122
I20260620 00:15:49.183156 16710 tablet_service.cc:1514] Processing DeleteTablet for tablet c220d5b278dc46d38b14218abac51a1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:15:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58970
I20260620 00:15:49.183187 16515 tablet_replica.cc:330] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: stopping tablet replica
I20260620 00:15:49.183238 16515 raft_consensus.cc:2229] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:49.183297 16515 raft_consensus.cc:2258] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:49.183475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16356
I20260620 00:15:49.183640 16807 tablet_replica.cc:330] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: stopping tablet replica
I20260620 00:15:49.183692 16515 ts_tablet_manager.cc:1905] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:49.183844 16806 tablet_replica.cc:330] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: stopping tablet replica
I20260620 00:15:49.183990 16807 raft_consensus.cc:2229] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:15:49.184157 16807 raft_consensus.cc:2258] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:49.184361 16806 raft_consensus.cc:2229] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:15:49.184499 16806 raft_consensus.cc:2258] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:15:49.184824 16807 ts_tablet_manager.cc:1905] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:49.184945 16806 ts_tablet_manager.cc:1905] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:15:49.186903 16807 ts_tablet_manager.cc:1918] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:49.186902 16806 ts_tablet_manager.cc:1918] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:15:49.186988 16806 log.cc:1196] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/c220d5b278dc46d38b14218abac51a1c
I20260620 00:15:49.187158 16807 log.cc:1196] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/c220d5b278dc46d38b14218abac51a1c
I20260620 00:15:49.187295 16806 ts_tablet_manager.cc:1939] T c220d5b278dc46d38b14218abac51a1c P 11b43df81e2c470096413f9b41dfb5ee: Deleting consensus metadata
I20260620 00:15:49.187633 16807 ts_tablet_manager.cc:1939] T c220d5b278dc46d38b14218abac51a1c P 21c3d686991648a4916a25f54b662e5f: Deleting consensus metadata
I20260620 00:15:49.187897 16303 catalog_manager.cc:4841] TS 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): tablet c220d5b278dc46d38b14218abac51a1c (table test-table [id=e09c27d7df7f4a61b6ef058b56dc789a]) successfully deleted
I20260620 00:15:49.188170 16304 catalog_manager.cc:4841] TS 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): tablet c220d5b278dc46d38b14218abac51a1c (table test-table [id=e09c27d7df7f4a61b6ef058b56dc789a]) successfully deleted
W20260620 00:15:49.191591 16302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45967 (error 108)
W20260620 00:15:49.192203 16513 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:15:49.192073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:44586)
0620 00:15:49.192116 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:15:49.192161 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 00:15:49.192443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16511
W20260620 00:15:49.192356 16302 catalog_manager.cc:4584] TS c42a692c1f7a42ed9eb73dd3dda335ae (127.0.98.1:45967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c220d5b278dc46d38b14218abac51a1c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection reset by peer (error 104)
I20260620 00:15:49.198550 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16647
I20260620 00:15:49.203644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16285
I20260620 00:16:09.209879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--webserver_interface=127.0.98.62
--webserver_port=36905
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:09.298748 16809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:09.299073 16809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:09.299124 16809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:09.300376 16809 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:16:09.300442 16809 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:16:09.300462 16809 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:16:09.300482 16809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:09.300509 16809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:09.300572 16809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:09.300618 16809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:09.300649 16809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:09.302232 16809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:09.302546 16809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:09.302789 16809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:09.308200 16817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.308107 16815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.308107 16814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.308271 16809 server_base.cc:1034] running on GCE node
I20260620 00:16:09.309114 16809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:09.309454 16809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:09.310607 16809 hybrid_clock.cc:648] HybridClock initialized: now 1781914569310585 us; error 35 us; skew 500 ppm
Jun 20 00:16:09 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:09.318121 16809 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:09.318674 16809 webserver.cc:460] Webserver started at http://127.0.98.62:36905/ using document root <none> and password file <none>
I20260620 00:16:09.318919 16809 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:09.318987 16809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:09.320524 16809 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:16:09.321353 16824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.321535 16809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:09.321643 16809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "2b42e7af16ef45bf9ace392979d9d3ca"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "cbe726acfcada7f590f34fe67563ffc5"
server_key_iv: "28e6889cc2a1495cf63bf3ae2db4aad7"
server_key_version: "encryptionkey@0"
I20260620 00:16:09.322058 16809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:09.333067 16809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:09.335546 16809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:09.335736 16809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:09.340312 16809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45367
I20260620 00:16:09.340386 16876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45367 every 8 connection(s)
I20260620 00:16:09.340893 16809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:09.343178 16877 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:16:09.343988 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap starting.
I20260620 00:16:09.345818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16809
I20260620 00:16:09.346202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45967
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=43907
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:16:09.353785 16877 log.cc:824] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:09.356109 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:16:09.356447 16877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Bootstrap complete.
I20260620 00:16:09.358776 16877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:16:09.359023 16877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Initialized, Role: FOLLOWER
I20260620 00:16:09.359218 16877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:16:09.359288 16877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:09.359359 16877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:09.359412 16877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:16:09.360654 16877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:16:09.360771 16877 leader_election.cc:302] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b42e7af16ef45bf9ace392979d9d3ca; no voters:
I20260620 00:16:09.360987 16877 leader_election.cc:288] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:16:09.361027 16881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:16:09.361142 16881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [term 3 LEADER]: Becoming Leader. State: Replica: 2b42e7af16ef45bf9ace392979d9d3ca, State: Running, Role: LEADER
I20260620 00:16:09.361248 16881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260620 00:16:09.361292 16877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:09.361537 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b42e7af16ef45bf9ace392979d9d3ca. Latest consensus state: current_term: 3 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:16:09.361608 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:09.361759 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b42e7af16ef45bf9ace392979d9d3ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260620 00:16:09.361836 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:09.362231 16885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:09.363179 16885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:09.363315 16885 catalog_manager.cc:1259] Loaded cluster ID: 559b5d16153e45f5a500f326fb41b34b
I20260620 00:16:09.363554 16885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:09.364881 16885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:09.365100 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Loaded TSK: 0
I20260620 00:16:09.365155 16885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b42e7af16ef45bf9ace392979d9d3ca: Loaded TSK: 1
F20260620 00:16:09.365401 16885 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb42237b9ad google::LogMessage::Fail() at ??:0
@ 0x7fb42237e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb42237b4de google::LogMessage::Flush() at ??:0
@ 0x7fb42237ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb424ae8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb424ae5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb42a2d91cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb42a2d948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb42a2d9594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb42a29fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb42a2a9ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb4242490d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb42423e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4250a26db start_thread at ??:0
@ 0x7fb420e7961f clone at ??:0
W20260620 00:16:09.465860 16879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:09.466207 16879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:09.466253 16879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:09.468533 16879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:09.468668 16879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:09.468719 16879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:09.471259 16879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43907
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:09.471599 16879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:09.471947 16879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:09.478412 16919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.478412 16918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.478643 16921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.478856 16879 server_base.cc:1034] running on GCE node
I20260620 00:16:09.479094 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:09.479405 16879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:09.480628 16879 hybrid_clock.cc:648] HybridClock initialized: now 1781914569480609 us; error 32 us; skew 500 ppm
Jun 20 00:16:09 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:09.483753 16879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:16:09.484366 16879 webserver.cc:460] Webserver started at http://127.0.98.1:43907/ using document root <none> and password file <none>
I20260620 00:16:09.484707 16879 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:09.484781 16879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:09.486487 16879 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:09.487421 16928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.487636 16879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:09.487759 16879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "c42a692c1f7a42ed9eb73dd3dda335ae"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "03d65c6c77ee5206a0b7388a575a55ae"
server_key_iv: "2663ca6cc1b0c69ffab62ac97b055f0a"
server_key_version: "encryptionkey@0"
I20260620 00:16:09.488168 16879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:09.499155 16879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:09.501701 16879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:09.501878 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:09.502149 16879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:09.502674 16935 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:16:09.504755 16879 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:16:09.504851 16879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:16:09.504897 16879 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:16:09.505650 16879 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:16:09.505709 16879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:16:09.505790 16935 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap starting.
I20260620 00:16:09.507967 16935 log.cc:824] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:09.512049 16935 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:16:09.512604 16935 tablet_bootstrap.cc:492] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Bootstrap complete.
I20260620 00:16:09.512830 16935 ts_tablet_manager.cc:1397] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20260620 00:16:09.513288 16879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45967
I20260620 00:16:09.513703 16879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:09.514258 17043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45967 every 8 connection(s)
I20260620 00:16:09.515039 16935 raft_consensus.cc:348] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:09.515352 16935 raft_consensus.cc:729] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a692c1f7a42ed9eb73dd3dda335ae, State: Initialized, Role: FOLLOWER
I20260620 00:16:09.515493 16935 consensus_queue.cc:260] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:09.515779 16935 ts_tablet_manager.cc:1428] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:16:09.522528 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 16879
I20260620 00:16:09.522907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38091
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=33453
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:09.596938 17046 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:16:09.514466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:39293)
0620 00:16:09.514652 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.514701 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:09.515008 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:09.596722 (+ 81714us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
W20260620 00:16:09.597280 17044 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45367 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45367: Client connection negotiation failed: client connection to 127.0.98.62:45367: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:16:09.597671 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.597530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:57673)
0620 00:16:09.597588 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.597639 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 00:16:09.598038 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.597916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:59181)
0620 00:16:09.597987 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.598016 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 00:16:09.598177 17044 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:16:09.609954 17048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:09.610190 17048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:09.610216 17048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:09.611475 17048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:09.611527 17048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:09.611557 17048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:09.613193 17048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33453
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:09.613473 17048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:09.613723 17048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:09.619102 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.619045 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.619289 17048 server_base.cc:1034] running on GCE node
W20260620 00:16:09.619006 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.619596 17048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:09.619870 17048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:09.621047 17048 hybrid_clock.cc:648] HybridClock initialized: now 1781914569621027 us; error 37 us; skew 500 ppm
Jun 20 00:16:09 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:09.623733 17048 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:16:09.624279 17048 webserver.cc:460] Webserver started at http://127.0.98.2:33453/ using document root <none> and password file <none>
I20260620 00:16:09.624511 17048 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:09.624615 17048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:09.626215 17048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:09.626888 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.627099 17048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:09.627200 17048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "11b43df81e2c470096413f9b41dfb5ee"
format_stamp: "Formatted at 2026-06-20 00:15:27 on dist-test-slave-28bv"
server_key: "7cb33f572b9637145cf8e62d5b198cde"
server_key_iv: "931756aaa6b8fe5669d0a4dc2f72b5c7"
server_key_version: "encryptionkey@0"
I20260620 00:16:09.627578 17048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:09.652045 17048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:09.654522 17048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:09.654649 17048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:09.654865 17048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:09.655118 17048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:09.655150 17048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.655174 17048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:09.655187 17048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.661098 17048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38091
I20260620 00:16:09.661193 17176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38091 every 8 connection(s)
I20260620 00:16:09.661531 17048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:09.662916 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.662254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:54345)
0620 00:16:09.662459 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.662538 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260620 00:16:09.663196 17177 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45367 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45367: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
I20260620 00:16:09.663676 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.663581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:52149)
0620 00:16:09.663626 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.663656 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 00:16:09.664027 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.663957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:44657)
0620 00:16:09.663991 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.664013 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 00:16:09.664155 17177 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:16:09.669157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17048
I20260620 00:16:09.669503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40563
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45367
--webserver_port=42815
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:09.760206 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:09.760456 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:09.760485 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:09.761826 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:09.761893 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:09.761924 17180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:09.763478 17180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42815
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:09.763722 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:09.763931 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:09.769336 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.769605 17180 server_base.cc:1034] running on GCE node
W20260620 00:16:09.769340 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:09.769336 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:09.770054 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:09.770364 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:09.771557 17180 hybrid_clock.cc:648] HybridClock initialized: now 1781914569771543 us; error 32 us; skew 500 ppm
Jun 20 00:16:09 dist-test-slave-28bv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:09.774530 17180 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:16:09.775100 17180 webserver.cc:460] Webserver started at http://127.0.98.3:42815/ using document root <none> and password file <none>
I20260620 00:16:09.775388 17180 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:09.775457 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:09.777069 17180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:16:09.777971 17195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.778290 17180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:09.778424 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "21c3d686991648a4916a25f54b662e5f"
format_stamp: "Formatted at 2026-06-20 00:15:28 on dist-test-slave-28bv"
server_key: "054adf0afea124c1cbeb24d4b7b5ab5f"
server_key_iv: "bd690b0b76f73208d1ee4b8c5f1ff27f"
server_key_version: "encryptionkey@0"
I20260620 00:16:09.778872 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:09.792919 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:09.795176 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:09.795408 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:09.795665 17180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:09.795984 17180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:09.796038 17180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.796080 17180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:09.796113 17180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:09.802523 17180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260620 00:16:09.802562 17308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260620 00:16:09.802943 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:09.804145 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.803662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:48177)
0620 00:16:09.803918 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.803998 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
W20260620 00:16:09.804656 17309 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45367 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45367: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
I20260620 00:16:09.804997 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.804874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:54069)
0620 00:16:09.804939 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.804973 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:16:09.805490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17180
I20260620 00:16:09.805614 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:09.805495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:39853)
0620 00:16:09.805553 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:09.805589 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 00:16:09.805719 17309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:16:10.598884 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:10.598609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:56335)
0620 00:16:10.598768 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:10.598817 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260620 00:16:10.664808 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:10.664532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:43643)
0620 00:16:10.664700 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:10.664748 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260620 00:16:10.806424 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:10.806120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:53985)
0620 00:16:10.806307 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:10.806361 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":65,"threads_started":1}
I20260620 00:16:11.093998 17316 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:11.094174 17316 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:11.094552 17316 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563), 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
Jun 20 00:16:11 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 20 00:16:11 dist-test-slave-28bv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 00:16:11.105717 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.094896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59089 (local address 127.0.98.2:38091)
0620 00:16:11.095111 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:11.095122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:11.095170 (+ 48us) server_negotiation.cc:407] Connection header received
0620 00:16:11.095777 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:11.095780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:11.095826 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:11.095979 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:11.097037 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.097913 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.098803 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.099212 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.099337 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:11.102152 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:11.102156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:11.102156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:11.102172 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:11.103858 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104380 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:11.104383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:11.104383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:11.104482 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104769 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:11.104770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:11.104770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:11.105043 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:11.105135 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:11.105450 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 00:16:11.105593 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":64,"threads_started":1}
I20260620 00:16:11.105726 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.094994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33379 (local address 127.0.98.3:40563)
0620 00:16:11.095126 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:11.095136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:11.095169 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:16:11.095783 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:11.095786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:11.095827 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:11.095985 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:11.096893 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.097778 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.098801 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.099216 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.099337 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:11.101889 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:11.101898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:11.101899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:11.101910 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:11.103829 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104579 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:11.104583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:11.104583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:11.104695 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104914 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:11.104915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:11.104915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:11.105169 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:11.105257 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:11.105469 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 00:16:11.105602 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260620 00:16:11.105712 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.094776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.98.1:33379)
0620 00:16:11.094836 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:11.094848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:11.094902 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:11.096013 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:11.096016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:11.096480 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:11.096772 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:11.096779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.097884 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.097885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:11.098698 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:11.098709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.099308 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.099309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:11.099445 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:11.100124 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:11.100128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:11.101743 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:11.103948 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:11.103960 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:11.103963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:11.104420 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:11.104775 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:11.104776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:11.104846 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:11.105249 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:11.105250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:11.105311 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 00:16:11.105390 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:16:11.105578 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260620 00:16:11.105834 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.094779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38091 (local address 127.0.98.1:59089)
0620 00:16:11.094933 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:11.094944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:11.094999 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:11.096013 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:11.096016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:11.096498 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:11.096924 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:11.096926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.098003 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.098004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:11.098698 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:11.098701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:11.099313 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:11.099314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:11.099447 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:11.100105 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:11.100114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:11.102037 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:11.104007 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:11.104009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:11.104237 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:11.104586 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:11.104587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:11.104587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:11.104700 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:11.105114 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:11.105121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:11.105272 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:16:11.105373 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:16:11.105660 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":136,"thread_start_us":75,"threads_started":1}
W20260620 00:16:11.106711 16931 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
W20260620 00:16:11.107317 16930 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
I20260620 00:16:11.107383 16930 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42a692c1f7a42ed9eb73dd3dda335ae; no voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f
I20260620 00:16:11.107503 17316 raft_consensus.cc:2726] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:16:11.599799 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.599610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:46431)
0620 00:16:11.599710 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:11.599756 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 00:16:11.665673 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.665428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:37489)
0620 00:16:11.665578 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:11.665625 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
I20260620 00:16:11.807265 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:11.806993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:55441)
0620 00:16:11.807172 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:11.807216 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260620 00:16:12.600726 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:12.600451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:47765)
0620 00:16:12.600626 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:12.600674 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260620 00:16:12.666611 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:12.666374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:42981)
0620 00:16:12.666509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:12.666556 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260620 00:16:12.808084 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:12.807818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:57691)
0620 00:16:12.808004 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:12.808035 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
I20260620 00:16:12.992368 17325 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:12.992475 17325 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:12.992739 17325 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563), 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
W20260620 00:16:12.993311 16930 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
W20260620 00:16:12.993382 16931 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
I20260620 00:16:12.993409 16930 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42a692c1f7a42ed9eb73dd3dda335ae; no voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f
I20260620 00:16:12.993481 17325 raft_consensus.cc:2726] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:16:13.601586 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:13.601346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:59447)
0620 00:16:13.601490 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:13.601537 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260620 00:16:13.667409 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:13.667189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:40621)
0620 00:16:13.667318 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:13.667362 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260620 00:16:13.809082 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:13.808840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:47231)
0620 00:16:13.808981 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:13.809031 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260620 00:16:14.602514 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:14.602258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:59021)
0620 00:16:14.602426 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:14.602467 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260620 00:16:14.668355 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:14.668081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:51307)
0620 00:16:14.668250 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:14.668301 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260620 00:16:14.752439 17331 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:14.752596 17331 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:14.752817 17331 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563), 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
W20260620 00:16:14.753218 16931 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
W20260620 00:16:14.753345 16930 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
I20260620 00:16:14.753402 16930 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42a692c1f7a42ed9eb73dd3dda335ae; no voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f
I20260620 00:16:14.753494 17331 raft_consensus.cc:2726] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:16:14.809901 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:14.809667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:46067)
0620 00:16:14.809804 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:14.809852 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260620 00:16:15.603434 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:15.603099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:49319)
0620 00:16:15.603281 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:15.603366 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260620 00:16:15.669163 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:15.668943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:33181)
0620 00:16:15.669069 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:15.669108 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 00:16:15.810738 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:15.810453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:37153)
0620 00:16:15.810632 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:15.810679 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260620 00:16:16.359493 17336 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:16.359654 17336 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:16.359861 17336 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563), 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
W20260620 00:16:16.360378 16930 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
W20260620 00:16:16.360484 16931 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
I20260620 00:16:16.360527 16931 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42a692c1f7a42ed9eb73dd3dda335ae; no voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f
I20260620 00:16:16.360636 17336 raft_consensus.cc:2726] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:16:16.604247 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:16.603975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:60439)
0620 00:16:16.604143 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:16.604198 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260620 00:16:16.669911 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:16.669691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:45333)
0620 00:16:16.669826 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:16.669863 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260620 00:16:16.811650 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:16.811364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:46087)
0620 00:16:16.811553 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:16.811601 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260620 00:16:17.605265 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:17.604943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:41339)
0620 00:16:17.605167 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:17.605217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260620 00:16:17.670739 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:17.670465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:50155)
0620 00:16:17.670633 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:17.670690 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260620 00:16:17.812477 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:17.812237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:44477)
0620 00:16:17.812391 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:17.812430 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260620 00:16:18.606293 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:18.606057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:51659)
0620 00:16:18.606199 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:18.606244 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260620 00:16:18.671689 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:18.671412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:38601)
0620 00:16:18.671591 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:18.671640 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260620 00:16:18.813280 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:18.813020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:46893)
0620 00:16:18.813173 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:18.813222 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260620 00:16:19.607307 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:19.607043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:46905)
0620 00:16:19.607209 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:19.607257 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260620 00:16:19.672650 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:19.672316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:39339)
0620 00:16:19.672478 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:19.672536 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":74,"threads_started":1}
I20260620 00:16:19.814067 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:19.813746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:39065)
0620 00:16:19.813916 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:19.814018 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 00:16:20.608239 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:20.607978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:41357)
0620 00:16:20.608144 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:20.608191 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":68,"threads_started":1}
I20260620 00:16:20.673434 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:20.673199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:57539)
0620 00:16:20.673342 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:20.673388 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 00:16:20.814812 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:20.814580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:54809)
0620 00:16:20.814715 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:20.814762 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260620 00:16:21.609272 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:21.608975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:53479)
0620 00:16:21.609166 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:21.609221 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 00:16:21.674353 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:21.674106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:53645)
0620 00:16:21.674256 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:21.674304 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260620 00:16:21.815584 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:21.815334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:48739)
0620 00:16:21.815494 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:21.815534 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260620 00:16:22.610270 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:22.609951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:33999)
0620 00:16:22.610148 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:22.610218 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260620 00:16:22.675326 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:22.675034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:59071)
0620 00:16:22.675217 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:22.675278 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260620 00:16:22.816846 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:22.816309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:55757)
0620 00:16:22.816485 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:22.816533 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260620 00:16:22.865023 17358 raft_consensus.cc:482] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:22.865127 17358 raft_consensus.cc:504] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21c3d686991648a4916a25f54b662e5f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } } peers { permanent_uuid: "11b43df81e2c470096413f9b41dfb5ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38091 } } peers { permanent_uuid: "c42a692c1f7a42ed9eb73dd3dda335ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } }
I20260620 00:16:22.865307 17358 leader_election.cc:288] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563), 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091)
W20260620 00:16:22.865702 16931 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21c3d686991648a4916a25f54b662e5f (127.0.98.3:40563): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
W20260620 00:16:22.865826 16930 leader_election.cc:341] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 11b43df81e2c470096413f9b41dfb5ee (127.0.98.2:38091): Not found: Tablet not found: c220d5b278dc46d38b14218abac51a1c
I20260620 00:16:22.865886 16930 leader_election.cc:302] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c42a692c1f7a42ed9eb73dd3dda335ae; no voters: 11b43df81e2c470096413f9b41dfb5ee, 21c3d686991648a4916a25f54b662e5f
I20260620 00:16:22.865957 17358 raft_consensus.cc:2726] T c220d5b278dc46d38b14218abac51a1c P c42a692c1f7a42ed9eb73dd3dda335ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:16:23.611146 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:23.610890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:34155)
0620 00:16:23.611055 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:23.611099 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260620 00:16:23.676353 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:23.676039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:45643)
0620 00:16:23.676232 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:23.676282 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260620 00:16:23.817703 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:23.817451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:51329)
0620 00:16:23.817598 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:23.817645 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260620 00:16:24.612254 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:24.611985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:57737)
0620 00:16:24.612157 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:24.612204 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260620 00:16:24.677397 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:24.677101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:39899)
0620 00:16:24.677267 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:24.677316 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260620 00:16:24.808169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 16879
I20260620 00:16:24.813745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17048
I20260620 00:16:24.818751 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:24.818422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:40857)
0620 00:16:24.818608 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:24.818665 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260620 00:16:24.820899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17180
2026-06-20T00:16:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58601 ms)
[----------] 30 tests from SecurityITest (262776 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:24 dist-test-slave-28bv krb5kdc[17367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:24 dist-test-slave-28bv krb5kdc[17367](info): set up 2 sockets
Jun 20 00:16:24 dist-test-slave-28bv krb5kdc[17367](info): commencing operation
krb5kdc: starting...
W20260620 00:16:25.912335 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:25 dist-test-slave-28bv krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:26.050141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35009
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:26.136201 17380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:26.136462 17380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:26.136488 17380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:26.138060 17380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:26.138163 17380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:26.138180 17380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:26.138192 17380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:26.138463 17380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:26.140524 17380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:26.140800 17380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:26.141021 17380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:26.146351 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.146421 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.146351 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.146534 17380 server_base.cc:1034] running on GCE node
I20260620 00:16:26.147047 17380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:26.147398 17380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:26.148597 17380 hybrid_clock.cc:648] HybridClock initialized: now 1781914586148593 us; error 38 us; skew 500 ppm
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:26.151368 17380 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:26.151947 17380 webserver.cc:460] Webserver started at http://127.0.98.62:34609/ using document root <none> and password file <none>
I20260620 00:16:26.152194 17380 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:26.152275 17380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:26.152434 17380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:26.153630 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7c0c37a871754629be9e2fba23f87d1e"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "f242b7b9802d2b6557602f19407dee71"
server_key_iv: "34b612b389884a879714b6764a507efe"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.154027 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c0c37a871754629be9e2fba23f87d1e"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "f242b7b9802d2b6557602f19407dee71"
server_key_iv: "34b612b389884a879714b6764a507efe"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.155562 17380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:16:26.156419 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.156639 17380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:26.156754 17380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7c0c37a871754629be9e2fba23f87d1e"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "f242b7b9802d2b6557602f19407dee71"
server_key_iv: "34b612b389884a879714b6764a507efe"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.156847 17380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:26.190802 17380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:26.193499 17380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:26.193698 17380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:26.198366 17380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35009
I20260620 00:16:26.198406 17447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35009 every 8 connection(s)
I20260620 00:16:26.198814 17380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:26.199807 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:26.202560 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: Bootstrap starting.
I20260620 00:16:26.203423 17448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:26.203799 17448 log.cc:824] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:26.204861 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: No bootstrap required, opened a new log
I20260620 00:16:26.206550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17380
I20260620 00:16:26.206678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:26.206908 392 external_mini_cluster.cc:1428] Setting key d8689d93aa07014f7d4a05336a57c45b
I20260620 00:16:26.207190 17448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } }
I20260620 00:16:26.207394 17448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:26.207439 17448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0c37a871754629be9e2fba23f87d1e, State: Initialized, Role: FOLLOWER
I20260620 00:16:26.207620 17448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } }
I20260620 00:16:26.207824 17448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:26.207870 17448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:26.207906 17448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:26.208702 17448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } }
I20260620 00:16:26.208847 17448 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0c37a871754629be9e2fba23f87d1e; no voters:
I20260620 00:16:26.209069 17448 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:26.209090 17453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:26.209278 17453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [term 1 LEADER]: Becoming Leader. State: Replica: 7c0c37a871754629be9e2fba23f87d1e, State: Running, Role: LEADER
I20260620 00:16:26.209450 17448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:26.209424 17453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } }
I20260620 00:16:26.209954 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0c37a871754629be9e2fba23f87d1e. Latest consensus state: current_term: 1 leader_uuid: "7c0c37a871754629be9e2fba23f87d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } } }
I20260620 00:16:26.210057 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:26.210338 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0c37a871754629be9e2fba23f87d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0c37a871754629be9e2fba23f87d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35009 } } }
I20260620 00:16:26.210662 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:26.210852 17460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:26.211418 17460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:26.213156 17460 catalog_manager.cc:1347] Generated new cluster ID: 8e29872f352948d4a1ef84e204089f35
I20260620 00:16:26.213215 17460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:26.219980 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.207517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.0.98.62:35009)
0620 00:16:26.207791 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.207801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.207835 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:16:26.208008 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.208016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.208076 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.208234 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:26.209158 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.210647 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.211904 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.212417 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.212592 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.215693 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:26.215698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:26.215699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:26.215714 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:26.218041 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.218504 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.218509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.218510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.218696 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.218913 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.218914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.218914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.219326 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:26.219498 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.219577 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.219737 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260620 00:16:26.221671 17460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:26.222420 17460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:26.227700 17460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: Generated new TSK 0
I20260620 00:16:26.227931 17460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:16:26.263847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:26.350880 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:26.351137 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:26.351167 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:26.352533 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:26.352649 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:26.352682 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:26.354380 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:26.354662 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:26.354998 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:26.360524 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.360771 17474 server_base.cc:1034] running on GCE node
W20260620 00:16:26.360514 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.360514 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.361141 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:26.361421 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:26.362607 17474 hybrid_clock.cc:648] HybridClock initialized: now 1781914586362565 us; error 52 us; skew 500 ppm
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:26.365804 17474 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:16:26.366312 17474 webserver.cc:460] Webserver started at http://127.0.98.1:40075/ using document root <none> and password file <none>
I20260620 00:16:26.366554 17474 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:26.366624 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:26.366799 17474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:26.367954 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "543ff2b2f9434f5aa896ef29a7408d22"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "dfcaf07118ec53de0fb18c4ccf056c60"
server_key_iv: "311753ade6e8e4b93a89e1941f50131d"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.368358 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "543ff2b2f9434f5aa896ef29a7408d22"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "dfcaf07118ec53de0fb18c4ccf056c60"
server_key_iv: "311753ade6e8e4b93a89e1941f50131d"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.369982 17474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:26.370904 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.371214 17474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:16:26.371345 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "543ff2b2f9434f5aa896ef29a7408d22"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "dfcaf07118ec53de0fb18c4ccf056c60"
server_key_iv: "311753ade6e8e4b93a89e1941f50131d"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.371436 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:26.382339 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:26.385031 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:26.385226 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:26.385490 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:26.385825 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:26.385883 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.385951 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:26.385999 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.392822 17474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46553
I20260620 00:16:26.392972 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46553 every 8 connection(s)
I20260620 00:16:26.393291 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:26.399323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17474
I20260620 00:16:26.399461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:26.399672 392 external_mini_cluster.cc:1428] Setting key f5e0da5b32c679f4259ba666e52f464a
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:26.405138 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.394187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60587 (local address 127.0.98.62:35009)
0620 00:16:26.394280 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.394282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.394476 (+ 194us) server_negotiation.cc:407] Connection header received
0620 00:16:26.395475 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.395476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.395492 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.395561 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:26.396291 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.396926 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.398007 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.398250 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.398315 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.401106 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:26.401108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:26.401109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:26.401118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:26.402865 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.403762 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.403764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.403764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.403838 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.404188 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.404189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.404190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.404365 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:26.404445 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.404870 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.404988 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 00:16:26.405171 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.394142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35009 (local address 127.0.98.1:60587)
0620 00:16:26.394333 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.394378 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.395359 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.395640 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.395643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.395926 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:26.396139 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.396143 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.397030 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.397031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.397889 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.397892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.398369 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.398370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.398493 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.399176 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:26.399187 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:26.400996 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:26.403040 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.403053 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.403056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.403628 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.403952 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.403953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.403953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.404098 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.404465 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:26.404473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:26.404704 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.404816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.404976 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260620 00:16:26.405836 17603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35009
I20260620 00:16:26.405983 17603 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:26.406208 17603 heartbeater.cc:511] Master 127.0.98.62:35009 requested a full tablet report, sending...
I20260620 00:16:26.406970 17412 ts_manager.cc:194] Registered new tserver with Master: 543ff2b2f9434f5aa896ef29a7408d22 (127.0.98.1:46553)
I20260620 00:16:26.407671 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60587
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:16:26.433058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:26.523186 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:26.523444 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:26.523470 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:26.524864 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:26.524924 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:26.524955 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:26.526676 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:26.526990 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:26.527285 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:26.533336 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.533638 17608 server_base.cc:1034] running on GCE node
W20260620 00:16:26.533285 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.533294 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.534075 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:26.534396 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:26.535595 17608 hybrid_clock.cc:648] HybridClock initialized: now 1781914586535580 us; error 34 us; skew 500 ppm
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:26.538959 17608 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:16:26.539548 17608 webserver.cc:460] Webserver started at http://127.0.98.2:35169/ using document root <none> and password file <none>
I20260620 00:16:26.539831 17608 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:26.539908 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:26.540072 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:26.541335 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe132f6b579c40efa165b23fc2e668d0"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "28c6124981b7ae1064b095a2abdd071a"
server_key_iv: "387883669e1023d0e3972313c0d96556"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.541747 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe132f6b579c40efa165b23fc2e668d0"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "28c6124981b7ae1064b095a2abdd071a"
server_key_iv: "387883669e1023d0e3972313c0d96556"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.543673 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:16:26.544857 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.545172 17608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:16:26.545339 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "fe132f6b579c40efa165b23fc2e668d0"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "28c6124981b7ae1064b095a2abdd071a"
server_key_iv: "387883669e1023d0e3972313c0d96556"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.545440 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:26.555104 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:26.557698 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:26.557881 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:26.558147 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:26.558632 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:26.558702 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.558743 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:26.558760 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.565763 17608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45109
I20260620 00:16:26.565867 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45109 every 8 connection(s)
I20260620 00:16:26.566241 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:26.569293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17608
I20260620 00:16:26.569402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:26.569628 392 external_mini_cluster.cc:1428] Setting key 02ec3863ab9d843a4e9abf8881f72d30
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:26.591488 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.567172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56003 (local address 127.0.98.62:35009)
0620 00:16:26.567285 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.567287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.567536 (+ 249us) server_negotiation.cc:407] Connection header received
0620 00:16:26.568815 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.568816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.568829 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.568893 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:26.569821 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.570675 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.572447 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.572840 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.572898 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.586838 (+ 13940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:26.586842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:26.586842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:26.586858 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:26.588938 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.589740 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.589742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.589743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.589875 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.590402 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.590406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.590406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.590645 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:26.590750 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.591210 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.591336 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 00:16:26.591602 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.567102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35009 (local address 127.0.98.2:56003)
0620 00:16:26.567286 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.567338 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.568641 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.569086 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.569090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.569435 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:26.569711 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.569717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.570803 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.570804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.572290 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.572295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.573016 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.573017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.573288 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.580803 (+ 7515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:26.580822 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:26.586596 (+ 5774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:26.589102 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.589114 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.589117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.589413 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.589976 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.589978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.589978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.590241 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.590798 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:26.590815 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:26.591000 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.591144 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.591353 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260620 00:16:26.592360 17737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35009
I20260620 00:16:26.592483 17737 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:26.592835 17737 heartbeater.cc:511] Master 127.0.98.62:35009 requested a full tablet report, sending...
I20260620 00:16:26.593382 17412 ts_manager.cc:194] Registered new tserver with Master: fe132f6b579c40efa165b23fc2e668d0 (127.0.98.2:45109)
I20260620 00:16:26.593824 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:16:26.608417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:26.693439 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:26.693729 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:26.693775 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:26.695191 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:26.695286 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:26.695351 17742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:26.697266 17742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:26.697616 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:26.697885 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:26.703730 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.703730 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:26.703730 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:26.704202 17742 server_base.cc:1034] running on GCE node
I20260620 00:16:26.704474 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:26.704789 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:26.705996 17742 hybrid_clock.cc:648] HybridClock initialized: now 1781914586705940 us; error 77 us; skew 500 ppm
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:26.708889 17742 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:16:26.709405 17742 webserver.cc:460] Webserver started at http://127.0.98.3:33689/ using document root <none> and password file <none>
I20260620 00:16:26.709679 17742 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:26.709750 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:26.709909 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:26.711112 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "3731a541b835498d84224049bd1b7984"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "3eaf323200c64fee58a85c999682852b"
server_key_iv: "7543b9011eac9e6e3bd725d123fa278a"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.711555 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3731a541b835498d84224049bd1b7984"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "3eaf323200c64fee58a85c999682852b"
server_key_iv: "7543b9011eac9e6e3bd725d123fa278a"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.713109 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:26.714007 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.714187 17742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:26.714274 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "3731a541b835498d84224049bd1b7984"
format_stamp: "Formatted at 2026-06-20 00:16:26 on dist-test-slave-28bv"
server_key: "3eaf323200c64fee58a85c999682852b"
server_key_iv: "7543b9011eac9e6e3bd725d123fa278a"
server_key_version: "encryptionkey@0"
I20260620 00:16:26.714336 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:26.728156 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:26.730547 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:26.730695 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:26.730912 17742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:26.731236 17742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:26.731273 17742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.731297 17742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:26.731346 17742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:26.737843 17742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35611
I20260620 00:16:26.737879 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35611 every 8 connection(s)
I20260620 00:16:26.738219 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:26.744508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17742
I20260620 00:16:26.744664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:26.744853 392 external_mini_cluster.cc:1428] Setting key 148518182aec65c4728276b3bca8af01
Jun 20 00:16:26 dist-test-slave-28bv krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:26.748644 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.739061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53625 (local address 127.0.98.62:35009)
0620 00:16:26.739110 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.739112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.739346 (+ 234us) server_negotiation.cc:407] Connection header received
0620 00:16:26.740252 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.740254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.740269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.740338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:26.741083 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.741604 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.742711 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.742960 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.743093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.745695 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:26.745697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:26.745697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:26.745705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:26.747010 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.747485 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.747486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.747487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.747554 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.747965 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.747966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.747966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.748084 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:26.748144 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.748420 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.748492 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:26.748911 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.738974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35009 (local address 127.0.98.3:53625)
0620 00:16:26.739162 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.739234 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.740150 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.740393 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.740396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.740758 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:26.740978 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.740982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.741715 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.741717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.742560 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.742562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.743110 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.743111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.743256 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.743793 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:26.743803 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:26.745576 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:26.747105 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.747115 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.747118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.747388 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.747747 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:26.747748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:26.747749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:26.747889 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:26.748150 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:26.748154 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:26.748299 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.748389 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.748530 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260620 00:16:26.749423 17871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35009
I20260620 00:16:26.749548 17871 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:26.749743 17871 heartbeater.cc:511] Master 127.0.98.62:35009 requested a full tablet report, sending...
I20260620 00:16:26.750281 17412 ts_manager.cc:194] Registered new tserver with Master: 3731a541b835498d84224049bd1b7984 (127.0.98.3:35611)
I20260620 00:16:26.750705 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53625
I20260620 00:16:26.757261 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:26.765815 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.758852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.62:35009)
0620 00:16:26.758991 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.758993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.759102 (+ 109us) server_negotiation.cc:407] Connection header received
0620 00:16:26.759235 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.759236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.759248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.759307 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:26.759897 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.760413 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.761245 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.761463 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.761558 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.762879 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:26.762881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:26.762882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:26.762889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:26.764328 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.764831 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.764833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.764833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.764900 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:26.765189 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:26.765190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:26.765190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:26.765370 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:26.765490 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.765596 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.765689 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 00:16:26.768183 17412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:26.770112 17412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:26.777860 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.98.2:45109)
0620 00:16:26.773450 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.773456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.773736 (+ 280us) server_negotiation.cc:407] Connection header received
0620 00:16:26.773775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.773777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.773790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.773877 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:26.774100 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775331 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.776638 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777405 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777485 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777548 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.777729 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.777775 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260620 00:16:26.777860 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45109 (local address 127.0.0.1:39360)
0620 00:16:26.773526 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.773571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.773643 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.773892 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.773895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.773898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:26.773998 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.774000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775472 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.775473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.776485 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.776488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777589 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.777656 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777660 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.777710 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.777769 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260620 00:16:26.777958 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.98.3:35611)
0620 00:16:26.773881 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.773886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.773896 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:26.773935 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.773937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.773952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.774171 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:26.774251 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775310 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.776808 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777335 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777518 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.777825 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.777872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":109,"threads_started":1}
I20260620 00:16:26.778510 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.98.1:46553)
0620 00:16:26.774021 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.774026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.774036 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:26.774072 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.774075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.774090 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.774162 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:26.774411 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775181 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.776800 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777540 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777607 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777660 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.777826 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.777870 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":76,"threads_started":1}
I20260620 00:16:26.778599 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35611 (local address 127.0.0.1:44056)
0620 00:16:26.773524 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.773571 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.773642 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.774059 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.774060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.774062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:26.774097 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.774098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775420 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.775421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.776698 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.776701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777454 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.777558 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777573 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.777667 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.777741 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":35,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260620 00:16:26.778724 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.773376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.0.1:43612)
0620 00:16:26.773524 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.773571 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.773642 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.774265 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.774266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.774268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:26.774323 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.774325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.775293 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.775294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.776698 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.776701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.777621 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.777622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.777685 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.777689 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.777736 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.777776 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260620 00:16:26.778792 17671 tablet_service.cc:1467] Processing CreateTablet for tablet 82c8f2842754471c850085465568d676 (DEFAULT_TABLE table=test-table [id=e596ce3c2a3541ce969617eaecd45cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:26.779223 17671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c8f2842754471c850085465568d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:26.779409 17805 tablet_service.cc:1467] Processing CreateTablet for tablet 82c8f2842754471c850085465568d676 (DEFAULT_TABLE table=test-table [id=e596ce3c2a3541ce969617eaecd45cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:26.779598 17537 tablet_service.cc:1467] Processing CreateTablet for tablet 82c8f2842754471c850085465568d676 (DEFAULT_TABLE table=test-table [id=e596ce3c2a3541ce969617eaecd45cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:26.779744 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c8f2842754471c850085465568d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:26.779953 17537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c8f2842754471c850085465568d676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:26.782903 17888 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: Bootstrap starting.
I20260620 00:16:26.783119 17889 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Bootstrap starting.
I20260620 00:16:26.783778 17889 tablet_bootstrap.cc:654] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:26.783778 17888 tablet_bootstrap.cc:654] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:26.784446 17889 log.cc:824] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:26.784446 17888 log.cc:824] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:26.784811 17890 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Bootstrap starting.
I20260620 00:16:26.785362 17889 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: No bootstrap required, opened a new log
I20260620 00:16:26.785444 17889 ts_tablet_manager.cc:1397] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:16:26.785526 17890 tablet_bootstrap.cc:654] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:26.785604 17888 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: No bootstrap required, opened a new log
I20260620 00:16:26.785672 17888 ts_tablet_manager.cc:1397] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:16:26.786085 17890 log.cc:824] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:26.786835 17890 tablet_bootstrap.cc:492] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: No bootstrap required, opened a new log
I20260620 00:16:26.786926 17890 ts_tablet_manager.cc:1397] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:16:26.787678 17889 raft_consensus.cc:348] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.787678 17888 raft_consensus.cc:348] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.787889 17889 raft_consensus.cc:374] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:26.787889 17888 raft_consensus.cc:374] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:26.787930 17888 raft_consensus.cc:729] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543ff2b2f9434f5aa896ef29a7408d22, State: Initialized, Role: FOLLOWER
I20260620 00:16:26.787930 17889 raft_consensus.cc:729] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe132f6b579c40efa165b23fc2e668d0, State: Initialized, Role: FOLLOWER
I20260620 00:16:26.788094 17888 consensus_queue.cc:260] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.788203 17889 consensus_queue.cc:260] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.788431 17888 ts_tablet_manager.cc:1428] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:16:26.788472 17889 ts_tablet_manager.cc:1428] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:16:26.788604 17737 heartbeater.cc:503] Master 127.0.98.62:35009 was elected leader, sending a full tablet report...
I20260620 00:16:26.788618 17603 heartbeater.cc:503] Master 127.0.98.62:35009 was elected leader, sending a full tablet report...
I20260620 00:16:26.788736 17890 raft_consensus.cc:348] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.788935 17890 raft_consensus.cc:374] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:26.788980 17890 raft_consensus.cc:729] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3731a541b835498d84224049bd1b7984, State: Initialized, Role: FOLLOWER
I20260620 00:16:26.789101 17890 consensus_queue.cc:260] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.789404 17890 ts_tablet_manager.cc:1428] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:16:26.789430 17871 heartbeater.cc:503] Master 127.0.98.62:35009 was elected leader, sending a full tablet report...
W20260620 00:16:26.816907 17738 tablet.cc:2369] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:26.835528 17895 raft_consensus.cc:482] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:26.835745 17895 raft_consensus.cc:504] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.836205 17895 leader_election.cc:288] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3731a541b835498d84224049bd1b7984 (127.0.98.3:35611), 543ff2b2f9434f5aa896ef29a7408d22 (127.0.98.1:46553)
I20260620 00:16:26.839419 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.836498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33133 (local address 127.0.98.3:35611)
0620 00:16:26.836589 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.836592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.836601 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:26.836632 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.836633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.836643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.836698 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:26.836887 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.837475 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.838543 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.838982 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.839042 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.839051 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.839253 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.839344 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:16:26.839416 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.836404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35611 (local address 127.0.98.2:33133)
0620 00:16:26.836481 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.836495 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.836572 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.836728 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.836729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.836731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:26.836804 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.836806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.837571 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.837572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.838423 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.838426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.839071 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.839072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.839168 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.839172 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.839227 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.839306 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 00:16:26.839493 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.836634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46773 (local address 127.0.98.1:46553)
0620 00:16:26.836688 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.836690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.836831 (+ 141us) server_negotiation.cc:407] Connection header received
0620 00:16:26.836862 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.836863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.836877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.836934 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:26.837104 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.837613 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.838663 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.839082 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.839153 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.839162 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.839366 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.839410 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:16:26.839538 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.836513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.98.2:46773)
0620 00:16:26.836726 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.836737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.836811 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.836959 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:26.836960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:26.836961 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:26.837029 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.837031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.837713 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.837714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.838556 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:26.838558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.839175 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.839176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:26.839273 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.839277 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:26.839339 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:16:26.839378 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":160,"threads_started":1}
I20260620 00:16:26.839875 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c8f2842754471c850085465568d676" candidate_uuid: "fe132f6b579c40efa165b23fc2e668d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3731a541b835498d84224049bd1b7984" is_pre_election: true
I20260620 00:16:26.839985 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c8f2842754471c850085465568d676" candidate_uuid: "fe132f6b579c40efa165b23fc2e668d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "543ff2b2f9434f5aa896ef29a7408d22" is_pre_election: true
I20260620 00:16:26.840052 17825 raft_consensus.cc:2454] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe132f6b579c40efa165b23fc2e668d0 in term 0.
I20260620 00:16:26.840189 17557 raft_consensus.cc:2454] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe132f6b579c40efa165b23fc2e668d0 in term 0.
I20260620 00:16:26.840353 17624 leader_election.cc:302] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3731a541b835498d84224049bd1b7984, fe132f6b579c40efa165b23fc2e668d0; no voters:
I20260620 00:16:26.840559 17895 raft_consensus.cc:2781] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:26.840634 17895 raft_consensus.cc:482] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:26.840670 17895 raft_consensus.cc:3037] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:26.841439 17895 raft_consensus.cc:504] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.841634 17895 leader_election.cc:288] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [CANDIDATE]: Term 1 election: Requested vote from peers 3731a541b835498d84224049bd1b7984 (127.0.98.3:35611), 543ff2b2f9434f5aa896ef29a7408d22 (127.0.98.1:46553)
I20260620 00:16:26.841864 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c8f2842754471c850085465568d676" candidate_uuid: "fe132f6b579c40efa165b23fc2e668d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3731a541b835498d84224049bd1b7984"
I20260620 00:16:26.841890 17557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c8f2842754471c850085465568d676" candidate_uuid: "fe132f6b579c40efa165b23fc2e668d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "543ff2b2f9434f5aa896ef29a7408d22"
I20260620 00:16:26.841988 17557 raft_consensus.cc:3037] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:26.841993 17825 raft_consensus.cc:3037] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:26.842818 17825 raft_consensus.cc:2454] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe132f6b579c40efa165b23fc2e668d0 in term 1.
I20260620 00:16:26.843027 17557 raft_consensus.cc:2454] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe132f6b579c40efa165b23fc2e668d0 in term 1.
I20260620 00:16:26.843042 17624 leader_election.cc:302] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3731a541b835498d84224049bd1b7984, fe132f6b579c40efa165b23fc2e668d0; no voters:
I20260620 00:16:26.843145 17895 raft_consensus.cc:2781] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:26.843329 17895 raft_consensus.cc:686] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 1 LEADER]: Becoming Leader. State: Replica: fe132f6b579c40efa165b23fc2e668d0, State: Running, Role: LEADER
I20260620 00:16:26.843456 17895 consensus_queue.cc:237] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } }
I20260620 00:16:26.844427 17411 catalog_manager.cc:5495] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe132f6b579c40efa165b23fc2e668d0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fe132f6b579c40efa165b23fc2e668d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe132f6b579c40efa165b23fc2e668d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45109 } health_report { overall_health: HEALTHY } } }
I20260620 00:16:26.856297 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.853597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.98.2:45109)
0620 00:16:26.853662 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:26.853664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:26.853672 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:16:26.853707 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:26.853707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:26.853728 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:26.853800 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:26.853939 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.854418 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.855291 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:26.855490 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:26.855558 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:26.855858 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:26.855968 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:26.856050 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:26.856180 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 00:16:26.856205 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:16:26.858490 17557 raft_consensus.cc:1261] T 82c8f2842754471c850085465568d676 P 543ff2b2f9434f5aa896ef29a7408d22 [term 1 FOLLOWER]: Refusing update from remote peer fe132f6b579c40efa165b23fc2e668d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:26.858494 17825 raft_consensus.cc:1261] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 1 FOLLOWER]: Refusing update from remote peer fe132f6b579c40efa165b23fc2e668d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:26.858912 17898 consensus_queue.cc:1035] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3731a541b835498d84224049bd1b7984" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:26.859088 17895 consensus_queue.cc:1035] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "543ff2b2f9434f5aa896ef29a7408d22" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:26.863294 17904 mvcc.cc:205] Tried to move back new op lower bound from 7298722147769073664 to 7298722147711569920. Current Snapshot: MvccSnapshot[applied={T|T < 7298722147769073664}]
I20260620 00:16:26.863548 17901 mvcc.cc:205] Tried to move back new op lower bound from 7298722147769073664 to 7298722147711569920. Current Snapshot: MvccSnapshot[applied={T|T < 7298722147769073664}]
I20260620 00:16:26.866000 17412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:26.866138 17412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:26.867161 17412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c0c37a871754629be9e2fba23f87d1e: Sending DeleteTablet for 3 replicas of tablet 82c8f2842754471c850085465568d676
I20260620 00:16:26.867554 17671 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c8f2842754471c850085465568d676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39360
I20260620 00:16:26.867682 17537 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c8f2842754471c850085465568d676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43612
I20260620 00:16:26.867808 17911 tablet_replica.cc:330] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: stopping tablet replica
I20260620 00:16:26.867908 17911 raft_consensus.cc:2229] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:16:26.867905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17474
I20260620 00:16:26.868045 17911 raft_consensus.cc:2258] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:26.868088 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet 82c8f2842754471c850085465568d676 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44056
I20260620 00:16:26.868512 17913 tablet_replica.cc:330] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: stopping tablet replica
I20260620 00:16:26.868651 17913 raft_consensus.cc:2229] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:26.868758 17913 raft_consensus.cc:2258] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:26.869277 17913 ts_tablet_manager.cc:1905] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:26.871238 17913 ts_tablet_manager.cc:1918] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:26.871346 17913 log.cc:1196] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/82c8f2842754471c850085465568d676
I20260620 00:16:26.871681 17913 ts_tablet_manager.cc:1939] T 82c8f2842754471c850085465568d676 P 3731a541b835498d84224049bd1b7984: Deleting consensus metadata
I20260620 00:16:26.872224 17396 catalog_manager.cc:4841] TS 3731a541b835498d84224049bd1b7984 (127.0.98.3:35611): tablet 82c8f2842754471c850085465568d676 (table test-table [id=e596ce3c2a3541ce969617eaecd45cd3]) successfully deleted
I20260620 00:16:26.872893 17911 ts_tablet_manager.cc:1905] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:26.875174 17911 ts_tablet_manager.cc:1918] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 00:16:26.875166 17396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46553 (error 108)
I20260620 00:16:26.875361 17911 log.cc:1196] T 82c8f2842754471c850085465568d676 P fe132f6b579c40efa165b23fc2e668d0: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/82c8f2842754471c850085465568d676
W20260620 00:16:26.875880 17882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:16:26.875664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.0.1:43616)
0620 00:16:26.875710 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.875723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:26.875777 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:26.875823 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46553: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 00:16:26.876039 17396 catalog_manager.cc:4584] TS 543ff2b2f9434f5aa896ef29a7408d22 (127.0.98.1:46553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82c8f2842754471c850085465568d676: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46553: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 00:16:26.876091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17608
I20260620 00:16:26.881263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17742
I20260620 00:16:26.881438 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:26.881253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45109 (local address 127.0.0.1:39376)
0620 00:16:26.881327 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:26.881388 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 00:16:26.886081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17380
2026-06-20T00:16:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2064 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T00:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:26Z Disabled control of system clock
I20260620 00:16:26.914479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44629
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:27.001792 17918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:27.002187 17918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:27.002239 17918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:27.003845 17918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:27.003904 17918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:27.003917 17918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:27.003930 17918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:27.003942 17918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:27.005457 17918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:27.005720 17918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:27.006024 17918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:27.012168 17924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.012168 17923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.012176 17926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:27.012210 17918 server_base.cc:1034] running on GCE node
I20260620 00:16:27.012749 17918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:27.013126 17918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:27.014317 17918 hybrid_clock.cc:648] HybridClock initialized: now 1781914587014284 us; error 40 us; skew 500 ppm
I20260620 00:16:27.015753 17918 webserver.cc:460] Webserver started at http://127.0.98.62:38615/ using document root <none> and password file <none>
I20260620 00:16:27.016072 17918 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:27.016148 17918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:27.016310 17918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:27.017568 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "f7b4843a20104ab59df8b2a24afb15a9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "058be8b57b4741531089e48223b254ce"
server_key_iv: "342adba49c88e7cba9fc22907825ed3a"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.017985 17918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7b4843a20104ab59df8b2a24afb15a9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "058be8b57b4741531089e48223b254ce"
server_key_iv: "342adba49c88e7cba9fc22907825ed3a"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.019569 17918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:27.020469 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.020720 17918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:27.020857 17918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "f7b4843a20104ab59df8b2a24afb15a9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "058be8b57b4741531089e48223b254ce"
server_key_iv: "342adba49c88e7cba9fc22907825ed3a"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.020948 17918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:27.052229 17918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:27.052649 17918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:27.052821 17918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:27.057196 17918 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44629
I20260620 00:16:27.057255 17984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44629 every 8 connection(s)
I20260620 00:16:27.057642 17918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:27.058660 17985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:27.059989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 17918
I20260620 00:16:27.060107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:27.060304 392 external_mini_cluster.cc:1428] Setting key 2fa1c29f516d6b793aa3cea809987ee4
I20260620 00:16:27.061890 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: Bootstrap starting.
I20260620 00:16:27.062680 17985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:27.063016 17985 log.cc:824] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:27.063849 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: No bootstrap required, opened a new log
I20260620 00:16:27.066345 17985 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } }
I20260620 00:16:27.066555 17985 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:27.066596 17985 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b4843a20104ab59df8b2a24afb15a9, State: Initialized, Role: FOLLOWER
I20260620 00:16:27.066771 17985 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } }
I20260620 00:16:27.066896 17985 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:27.066939 17985 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:27.066978 17985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:27.067044 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.060946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.0.98.62:44629)
0620 00:16:27.061190 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.061200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.061235 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:16:27.061793 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.061803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.061889 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.062046 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:27.062333 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.063679 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.064535 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.065215 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.065357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.066158 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:27.066162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:27.066162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:27.066391 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:27.066436 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:27.066453 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:27.066525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.066652 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.066751 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260620 00:16:27.067726 17985 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } }
I20260620 00:16:27.067859 17985 leader_election.cc:302] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b4843a20104ab59df8b2a24afb15a9; no voters:
I20260620 00:16:27.068050 17985 leader_election.cc:288] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:27.068111 17990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:27.068343 17990 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [term 1 LEADER]: Becoming Leader. State: Replica: f7b4843a20104ab59df8b2a24afb15a9, State: Running, Role: LEADER
I20260620 00:16:27.068356 17985 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:27.068508 17990 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } }
I20260620 00:16:27.068977 17991 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b4843a20104ab59df8b2a24afb15a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } } }
I20260620 00:16:27.069070 17991 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:27.069304 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b4843a20104ab59df8b2a24afb15a9. Latest consensus state: current_term: 1 leader_uuid: "f7b4843a20104ab59df8b2a24afb15a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b4843a20104ab59df8b2a24afb15a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44629 } } }
I20260620 00:16:27.069417 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:27.069342 17999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:27.070034 17999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:27.071976 17999 catalog_manager.cc:1347] Generated new cluster ID: adfbbe78f5544ceebea797704cb36b6c
I20260620 00:16:27.072059 17999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:27.081902 17999 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:27.082526 17999 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:27.086197 17999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: Generated new TSK 0
I20260620 00:16:27.086429 17999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:27.090848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--builtin_ntp_servers=127.0.98.20:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:27.182610 18009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:27.182869 18009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:27.182896 18009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:27.184194 18009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:27.184247 18009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:27.184274 18009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:27.185907 18009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:27.186182 18009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:27.186535 18009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:27.193368 18014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.193373 18015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.193555 18017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:27.193434 18009 server_base.cc:1034] running on GCE node
I20260620 00:16:27.193885 18009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:27.194223 18009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:27.195446 18009 hybrid_clock.cc:648] HybridClock initialized: now 1781914587195414 us; error 44 us; skew 500 ppm
I20260620 00:16:27.197347 18009 webserver.cc:460] Webserver started at http://127.0.98.1:45533/ using document root <none> and password file <none>
I20260620 00:16:27.197645 18009 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:27.197713 18009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:27.197840 18009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:27.199313 18009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "b5877226937245198cac03f2d336977a"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "4b263206fc72ff44e365ef4d29f78651"
server_key_iv: "1d884728dacedb831b157b064c39dc35"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.199733 18009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b5877226937245198cac03f2d336977a"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "4b263206fc72ff44e365ef4d29f78651"
server_key_iv: "1d884728dacedb831b157b064c39dc35"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.201579 18009 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 00:16:27.202761 18023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.203061 18009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:27.203169 18009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "b5877226937245198cac03f2d336977a"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "4b263206fc72ff44e365ef4d29f78651"
server_key_iv: "1d884728dacedb831b157b064c39dc35"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.203259 18009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:27.234056 18009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:27.234488 18009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:27.234614 18009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:27.234827 18009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:27.235181 18009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:27.235217 18009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.235241 18009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:27.235253 18009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.241328 18009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40031
I20260620 00:16:27.241366 18136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40031 every 8 connection(s)
I20260620 00:16:27.241734 18009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:27.247051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18009
I20260620 00:16:27.247184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:27.247393 392 external_mini_cluster.cc:1428] Setting key 610c182cd658d56ec94fc56703ddac7b
I20260620 00:16:27.247422 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.242554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57101 (local address 127.0.98.62:44629)
0620 00:16:27.242692 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.242695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.242904 (+ 209us) server_negotiation.cc:407] Connection header received
0620 00:16:27.243502 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.243504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.243518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.243574 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:27.244299 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.244886 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.245894 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.246093 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.246176 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.247090 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:27.247091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:27.247092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:27.247097 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:27.247121 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:27.247125 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:27.247199 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.247299 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.247339 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:16:27.247674 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.242549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44629 (local address 127.0.98.1:57101)
0620 00:16:27.242780 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.242815 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.243376 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.243661 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.243667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.243677 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:27.244204 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.244209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.244987 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.244988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.245793 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.245795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.246185 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.246187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.246334 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.246905 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:27.246906 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:27.246935 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:27.246960 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:27.247199 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:27.247207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:27.247210 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.247303 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.247415 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260620 00:16:27.248260 18137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44629
I20260620 00:16:27.248359 18137 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:27.248618 18137 heartbeater.cc:511] Master 127.0.98.62:44629 requested a full tablet report, sending...
I20260620 00:16:27.248636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--builtin_ntp_servers=127.0.98.20:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:16:27.250263 17948 ts_manager.cc:194] Registered new tserver with Master: b5877226937245198cac03f2d336977a (127.0.98.1:40031)
I20260620 00:16:27.251145 17948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57101
W20260620 00:16:27.334960 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:27.335219 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:27.335246 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:27.336647 18140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:27.336700 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:27.336730 18140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:27.338191 18140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:27.338418 18140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:27.338762 18140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:27.344054 18145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.344080 18146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.344054 18148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:27.344347 18140 server_base.cc:1034] running on GCE node
I20260620 00:16:27.344645 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:27.344983 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:27.346161 18140 hybrid_clock.cc:648] HybridClock initialized: now 1781914587346150 us; error 38 us; skew 500 ppm
I20260620 00:16:27.347783 18140 webserver.cc:460] Webserver started at http://127.0.98.2:35299/ using document root <none> and password file <none>
I20260620 00:16:27.348065 18140 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:27.348141 18140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:27.348304 18140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:27.349524 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c84cd0857df4b75ae959a38b8132eaf"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "151c34a82f5313bb265c270b7bff207c"
server_key_iv: "394008e76a91225ac69dbae53a0186f0"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.349922 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c84cd0857df4b75ae959a38b8132eaf"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "151c34a82f5313bb265c270b7bff207c"
server_key_iv: "394008e76a91225ac69dbae53a0186f0"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.351404 18140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:27.352342 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.352627 18140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:27.352785 18140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "9c84cd0857df4b75ae959a38b8132eaf"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "151c34a82f5313bb265c270b7bff207c"
server_key_iv: "394008e76a91225ac69dbae53a0186f0"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.352882 18140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:27.380453 18140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:27.380906 18140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:27.381094 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:27.381378 18140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:27.381779 18140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:27.381839 18140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.381876 18140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:27.381903 18140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.388500 18140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36781
I20260620 00:16:27.388587 18267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36781 every 8 connection(s)
I20260620 00:16:27.389012 18140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:27.394902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18140
I20260620 00:16:27.395056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:27.395258 392 external_mini_cluster.cc:1428] Setting key 3f361e82057939910c760d2151d50a56
I20260620 00:16:27.395846 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.389981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58127 (local address 127.0.98.62:44629)
0620 00:16:27.390064 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.390066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.390411 (+ 345us) server_negotiation.cc:407] Connection header received
0620 00:16:27.391025 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.391026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.391039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.391110 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:27.392003 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.392533 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.394006 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.394223 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.394293 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.395469 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:27.395471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:27.395472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:27.395477 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:27.395498 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:27.395504 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:27.395568 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.395732 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.395765 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:16:27.396136 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.389901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44629 (local address 127.0.98.2:58127)
0620 00:16:27.390195 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.390243 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.390786 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.391274 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.391279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.391289 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:27.391879 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.391884 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.392981 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.392982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.393902 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.393905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.394319 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.394319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.394451 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.395266 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:27.395266 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:27.395329 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:27.395368 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:27.395585 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:27.395593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:27.395597 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.395711 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.395852 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260620 00:16:27.396798 18268 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44629
I20260620 00:16:27.396849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--builtin_ntp_servers=127.0.98.20:38715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:16:27.396934 18268 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:27.397168 18268 heartbeater.cc:511] Master 127.0.98.62:44629 requested a full tablet report, sending...
I20260620 00:16:27.397640 17948 ts_manager.cc:194] Registered new tserver with Master: 9c84cd0857df4b75ae959a38b8132eaf (127.0.98.2:36781)
I20260620 00:16:27.398041 17948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58127
W20260620 00:16:27.486366 18271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:27.486613 18271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:27.486639 18271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:27.488051 18271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:27.488116 18271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:27.488147 18271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:27.489784 18271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:27.490113 18271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:27.490478 18271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:27.496275 18276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:27.496259 18277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:27.496335 18271 server_base.cc:1034] running on GCE node
W20260620 00:16:27.496251 18279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:27.496799 18271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:27.497051 18271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:27.498210 18271 hybrid_clock.cc:648] HybridClock initialized: now 1781914587498199 us; error 31 us; skew 500 ppm
I20260620 00:16:27.499644 18271 webserver.cc:460] Webserver started at http://127.0.98.3:40643/ using document root <none> and password file <none>
I20260620 00:16:27.499958 18271 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:27.500037 18271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:27.500195 18271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:27.501487 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "e718d25062a84491acb91c9cd32be7f9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "53d1922cc8d15aa23bc5907b766ae010"
server_key_iv: "f5f65527403e59df798410d6af3fae80"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.501893 18271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e718d25062a84491acb91c9cd32be7f9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "53d1922cc8d15aa23bc5907b766ae010"
server_key_iv: "f5f65527403e59df798410d6af3fae80"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.503387 18271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:16:27.504245 18285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.504444 18271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:27.504570 18271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "e718d25062a84491acb91c9cd32be7f9"
format_stamp: "Formatted at 2026-06-20 00:16:27 on dist-test-slave-28bv"
server_key: "53d1922cc8d15aa23bc5907b766ae010"
server_key_iv: "f5f65527403e59df798410d6af3fae80"
server_key_version: "encryptionkey@0"
I20260620 00:16:27.504673 18271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:27.537758 18271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:27.538161 18271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:27.538372 18271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:27.538651 18271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:27.539038 18271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:27.539105 18271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.539153 18271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:27.539179 18271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:27.546020 18271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35463
I20260620 00:16:27.546197 18398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35463 every 8 connection(s)
I20260620 00:16:27.546501 18271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:27.552444 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.547437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39401 (local address 127.0.98.62:44629)
0620 00:16:27.547520 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.547521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.547815 (+ 294us) server_negotiation.cc:407] Connection header received
0620 00:16:27.548334 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.548336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.548349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.548408 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:27.549225 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.549733 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.550845 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.551047 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.551127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.552076 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:27.552079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:27.552079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:27.552085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:27.552103 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:27.552115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:27.552174 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.552322 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.552353 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:16:27.552870 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.547416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44629 (local address 127.0.98.3:39401)
0620 00:16:27.547637 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.547673 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.548211 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.548490 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.548493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.548501 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:27.549114 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.549119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.549871 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.549872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.550732 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.550734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.551134 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.551136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.551289 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.551873 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:27.551873 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:27.551905 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:27.551962 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:27.552189 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:27.552196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:27.552199 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.552319 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.552435 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260620 00:16:27.553474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18271
I20260620 00:16:27.553540 18399 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44629
I20260620 00:16:27.553606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:27.553655 18399 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:27.553881 18399 heartbeater.cc:511] Master 127.0.98.62:44629 requested a full tablet report, sending...
I20260620 00:16:27.553897 392 external_mini_cluster.cc:1428] Setting key 79fbb806e2fb708811efba515c40ca3a
I20260620 00:16:27.554312 17949 ts_manager.cc:194] Registered new tserver with Master: e718d25062a84491acb91c9cd32be7f9 (127.0.98.3:35463)
I20260620 00:16:27.554824 17949 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39401
I20260620 00:16:27.554888 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:27.559566 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.556482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.0.98.62:44629)
0620 00:16:27.556588 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.556590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.556664 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:16:27.556729 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.556730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.556739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.556786 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:27.556939 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.557406 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.558204 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.558407 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.558492 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.559292 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:27.559293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:27.559293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:27.559299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:27.559317 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:27.559321 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:27.559368 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.559481 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.559502 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:16:27.561873 17949 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:27.563545 17949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:27.571133 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.98.2:36781)
0620 00:16:27.566730 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.566735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.566861 (+ 126us) server_negotiation.cc:407] Connection header received
0620 00:16:27.566996 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.566999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.567027 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.567104 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:27.567400 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.568651 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.569973 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.570703 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.570808 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.570911 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.570997 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.571037 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260620 00:16:27.571121 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36781 (local address 127.0.0.1:40980)
0620 00:16:27.566755 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.566801 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.566883 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.567238 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.567241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.567243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:27.567314 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.567316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.568793 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.568794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.569883 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.569886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.570812 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.570813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.570890 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.570905 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.570982 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.571037 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":68,"threads_started":1}
I20260620 00:16:27.571316 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35463 (local address 127.0.0.1:44206)
0620 00:16:27.567013 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.567023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.567074 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.567338 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.567339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.567342 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:27.567405 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.567407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.568580 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.568581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.569969 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.569972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.571046 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.571047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.571111 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.571115 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.571169 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.571217 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":3,"thread_start_us":70,"threads_started":1}
I20260620 00:16:27.571554 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40031 (local address 127.0.0.1:50752)
0620 00:16:27.567140 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.567148 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.567190 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.567610 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.567611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.567613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:27.567648 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.567649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.569147 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.569148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.570297 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.570300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.571325 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.571326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.571388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.571392 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.571448 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.571493 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260620 00:16:27.571660 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.0.98.3:35463)
0620 00:16:27.566822 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.566826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.567091 (+ 265us) server_negotiation.cc:407] Connection header received
0620 00:16:27.567145 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.567147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.567200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.567339 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:27.567485 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.568467 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.570101 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.570854 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.570925 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.571007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.571513 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.571557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":96,"threads_started":1}
I20260620 00:16:27.572111 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.566386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.0.98.1:40031)
0620 00:16:27.566727 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.566731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.567260 (+ 529us) server_negotiation.cc:407] Connection header received
0620 00:16:27.567499 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.567502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.567518 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.567599 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:27.567711 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.568875 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.570396 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.571153 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.571211 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.571290 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.571969 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.572022 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":84,"threads_started":1}
I20260620 00:16:27.572463 18333 tablet_service.cc:1467] Processing CreateTablet for tablet a84be1109649492381741d8285611bec (DEFAULT_TABLE table=test-table [id=3abeb059eccd4f29bde896eb69c97e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:27.572458 18202 tablet_service.cc:1467] Processing CreateTablet for tablet a84be1109649492381741d8285611bec (DEFAULT_TABLE table=test-table [id=3abeb059eccd4f29bde896eb69c97e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:27.572876 18202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84be1109649492381741d8285611bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:27.572876 18333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84be1109649492381741d8285611bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:27.572790 18071 tablet_service.cc:1467] Processing CreateTablet for tablet a84be1109649492381741d8285611bec (DEFAULT_TABLE table=test-table [id=3abeb059eccd4f29bde896eb69c97e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:27.573119 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a84be1109649492381741d8285611bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:27.575942 18413 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Bootstrap starting.
I20260620 00:16:27.576615 18414 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Bootstrap starting.
I20260620 00:16:27.576838 18413 tablet_bootstrap.cc:654] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:27.577157 18413 log.cc:824] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:27.577310 18414 tablet_bootstrap.cc:654] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:27.577605 18414 log.cc:824] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:27.577932 18415 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Bootstrap starting.
I20260620 00:16:27.578372 18414 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: No bootstrap required, opened a new log
I20260620 00:16:27.578462 18414 ts_tablet_manager.cc:1397] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:16:27.578640 18413 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: No bootstrap required, opened a new log
I20260620 00:16:27.578702 18413 ts_tablet_manager.cc:1397] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:16:27.580876 18413 raft_consensus.cc:348] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.581029 18413 raft_consensus.cc:374] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:27.581053 18413 raft_consensus.cc:729] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5877226937245198cac03f2d336977a, State: Initialized, Role: FOLLOWER
I20260620 00:16:27.580753 18414 raft_consensus.cc:348] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.581207 18414 raft_consensus.cc:374] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:27.581290 18414 raft_consensus.cc:729] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c84cd0857df4b75ae959a38b8132eaf, State: Initialized, Role: FOLLOWER
I20260620 00:16:27.581180 18413 consensus_queue.cc:260] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.581424 18414 consensus_queue.cc:260] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.581648 18413 ts_tablet_manager.cc:1428] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:16:27.581686 18137 heartbeater.cc:503] Master 127.0.98.62:44629 was elected leader, sending a full tablet report...
I20260620 00:16:27.581744 18268 heartbeater.cc:503] Master 127.0.98.62:44629 was elected leader, sending a full tablet report...
I20260620 00:16:27.582494 18414 ts_tablet_manager.cc:1428] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 00:16:27.583171 18415 tablet_bootstrap.cc:654] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:27.583480 18415 log.cc:824] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:27.584367 18415 tablet_bootstrap.cc:492] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: No bootstrap required, opened a new log
I20260620 00:16:27.584477 18415 ts_tablet_manager.cc:1397] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 00:16:27.586575 18415 raft_consensus.cc:348] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.586776 18415 raft_consensus.cc:374] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:27.586827 18415 raft_consensus.cc:729] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e718d25062a84491acb91c9cd32be7f9, State: Initialized, Role: FOLLOWER
I20260620 00:16:27.586967 18415 consensus_queue.cc:260] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.587288 18415 ts_tablet_manager.cc:1428] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:16:27.587494 18399 heartbeater.cc:503] Master 127.0.98.62:44629 was elected leader, sending a full tablet report...
W20260620 00:16:27.639706 18269 tablet.cc:2369] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:16:27.742558 18138 tablet.cc:2369] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:27.746807 18421 raft_consensus.cc:482] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:27.746928 18421 raft_consensus.cc:504] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.747375 18421 leader_election.cc:288] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c84cd0857df4b75ae959a38b8132eaf (127.0.98.2:36781), b5877226937245198cac03f2d336977a (127.0.98.1:40031)
I20260620 00:16:27.751196 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.747621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33839 (local address 127.0.98.2:36781)
0620 00:16:27.747680 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.747682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.747697 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:16:27.747733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.747734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.747747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.747818 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:27.747972 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.748813 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750245 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.750840 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750901 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.750909 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.751084 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.751125 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:16:27.751204 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.747727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41553 (local address 127.0.98.1:40031)
0620 00:16:27.747787 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.747789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.747855 (+ 66us) server_negotiation.cc:407] Connection header received
0620 00:16:27.748057 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.748058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.748071 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.748132 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:27.748276 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.748965 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750247 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.750801 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.750895 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.751084 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.751124 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:16:27.751199 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.747550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36781 (local address 127.0.98.3:33839)
0620 00:16:27.747606 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.747619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.747670 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.747825 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.747827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.747829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:27.747900 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.747902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.748904 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.748904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.750127 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.750130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.750917 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.750998 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.751004 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.751062 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.751108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":8}
I20260620 00:16:27.751204 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.747603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40031 (local address 127.0.98.3:41553)
0620 00:16:27.747785 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.747795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:27.747979 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:27.748152 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:27.748152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:27.748154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:27.748205 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.748206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.749030 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.749031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.750170 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:27.750172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.750911 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.750912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:27.750998 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.751004 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:16:27.751062 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:16:27.751131 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":7,"thread_start_us":108,"threads_started":1}
I20260620 00:16:27.751600 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84be1109649492381741d8285611bec" candidate_uuid: "e718d25062a84491acb91c9cd32be7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5877226937245198cac03f2d336977a" is_pre_election: true
I20260620 00:16:27.751618 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84be1109649492381741d8285611bec" candidate_uuid: "e718d25062a84491acb91c9cd32be7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c84cd0857df4b75ae959a38b8132eaf" is_pre_election: true
I20260620 00:16:27.751789 18222 raft_consensus.cc:2454] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e718d25062a84491acb91c9cd32be7f9 in term 0.
I20260620 00:16:27.751785 18091 raft_consensus.cc:2454] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e718d25062a84491acb91c9cd32be7f9 in term 0.
I20260620 00:16:27.752017 18288 leader_election.cc:302] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c84cd0857df4b75ae959a38b8132eaf, e718d25062a84491acb91c9cd32be7f9; no voters:
I20260620 00:16:27.752174 18421 raft_consensus.cc:2781] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:27.752230 18421 raft_consensus.cc:482] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:27.752256 18421 raft_consensus.cc:3037] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:27.753041 18421 raft_consensus.cc:504] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.753199 18421 leader_election.cc:288] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [CANDIDATE]: Term 1 election: Requested vote from peers 9c84cd0857df4b75ae959a38b8132eaf (127.0.98.2:36781), b5877226937245198cac03f2d336977a (127.0.98.1:40031)
I20260620 00:16:27.753333 18222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84be1109649492381741d8285611bec" candidate_uuid: "e718d25062a84491acb91c9cd32be7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c84cd0857df4b75ae959a38b8132eaf"
I20260620 00:16:27.753337 18091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a84be1109649492381741d8285611bec" candidate_uuid: "e718d25062a84491acb91c9cd32be7f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5877226937245198cac03f2d336977a"
I20260620 00:16:27.753422 18222 raft_consensus.cc:3037] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:27.753448 18091 raft_consensus.cc:3037] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:27.754168 18091 raft_consensus.cc:2454] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e718d25062a84491acb91c9cd32be7f9 in term 1.
I20260620 00:16:27.754246 18222 raft_consensus.cc:2454] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e718d25062a84491acb91c9cd32be7f9 in term 1.
I20260620 00:16:27.754344 18287 leader_election.cc:302] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5877226937245198cac03f2d336977a, e718d25062a84491acb91c9cd32be7f9; no voters:
I20260620 00:16:27.754449 18421 raft_consensus.cc:2781] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:27.754609 18421 raft_consensus.cc:686] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 1 LEADER]: Becoming Leader. State: Replica: e718d25062a84491acb91c9cd32be7f9, State: Running, Role: LEADER
I20260620 00:16:27.754724 18421 consensus_queue.cc:237] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } }
I20260620 00:16:27.755702 17947 catalog_manager.cc:5495] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e718d25062a84491acb91c9cd32be7f9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e718d25062a84491acb91c9cd32be7f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e718d25062a84491acb91c9cd32be7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35463 } health_report { overall_health: HEALTHY } } }
W20260620 00:16:27.797385 18400 tablet.cc:2369] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:27.803292 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.800368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.0.98.3:35463)
0620 00:16:27.800434 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:27.800436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:27.800444 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:16:27.800491 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:27.800492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:27.800510 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:27.800603 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:27.800879 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.801373 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.802285 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:27.802496 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:27.802562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:27.802776 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:27.802892 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:27.802971 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:27.803072 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 00:16:27.803116 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:16:27.805374 18222 raft_consensus.cc:1261] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 1 FOLLOWER]: Refusing update from remote peer e718d25062a84491acb91c9cd32be7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:27.805395 18091 raft_consensus.cc:1261] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 1 FOLLOWER]: Refusing update from remote peer e718d25062a84491acb91c9cd32be7f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:27.805794 18421 consensus_queue.cc:1035] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c84cd0857df4b75ae959a38b8132eaf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:27.805953 18423 consensus_queue.cc:1035] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5877226937245198cac03f2d336977a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:27.808480 18426 mvcc.cc:205] Tried to move back new op lower bound from 7298722151647510528 to 7298722151444213760. Current Snapshot: MvccSnapshot[applied={T|T < 7298722151647510528}]
I20260620 00:16:27.809679 18427 mvcc.cc:205] Tried to move back new op lower bound from 7298722151647510528 to 7298722151444213760. Current Snapshot: MvccSnapshot[applied={T|T < 7298722151647510528 or (T in {7298722151647510528})}]
I20260620 00:16:27.811838 17947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:27.811944 17947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:27.812757 17947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7b4843a20104ab59df8b2a24afb15a9: Sending DeleteTablet for 3 replicas of tablet a84be1109649492381741d8285611bec
I20260620 00:16:27.813122 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet a84be1109649492381741d8285611bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:27 UTC) from {username='slave'} at 127.0.0.1:50752
I20260620 00:16:27.813151 18333 tablet_service.cc:1514] Processing DeleteTablet for tablet a84be1109649492381741d8285611bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:27 UTC) from {username='slave'} at 127.0.0.1:44206
I20260620 00:16:27.813347 18202 tablet_service.cc:1514] Processing DeleteTablet for tablet a84be1109649492381741d8285611bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:27 UTC) from {username='slave'} at 127.0.0.1:40980
I20260620 00:16:27.813575 18436 tablet_replica.cc:330] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a: stopping tablet replica
I20260620 00:16:27.813563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18009
I20260620 00:16:27.813733 18436 raft_consensus.cc:2229] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:27.813918 18436 raft_consensus.cc:2258] T a84be1109649492381741d8285611bec P b5877226937245198cac03f2d336977a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:27.814208 18435 tablet_replica.cc:330] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: stopping tablet replica
I20260620 00:16:27.814273 18437 tablet_replica.cc:330] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: stopping tablet replica
I20260620 00:16:27.814390 18435 raft_consensus.cc:2229] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:16:27.814447 18437 raft_consensus.cc:2229] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:27.814553 18435 raft_consensus.cc:2258] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:27.814644 18437 raft_consensus.cc:2258] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:27.816619 18435 ts_tablet_manager.cc:1905] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:27.817121 18437 ts_tablet_manager.cc:1905] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:27.818851 18437 ts_tablet_manager.cc:1918] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:27.818854 18435 ts_tablet_manager.cc:1918] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:27.818931 18435 log.cc:1196] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/a84be1109649492381741d8285611bec
I20260620 00:16:27.819016 18437 log.cc:1196] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/a84be1109649492381741d8285611bec
I20260620 00:16:27.819319 18435 ts_tablet_manager.cc:1939] T a84be1109649492381741d8285611bec P e718d25062a84491acb91c9cd32be7f9: Deleting consensus metadata
I20260620 00:16:27.819399 18437 ts_tablet_manager.cc:1939] T a84be1109649492381741d8285611bec P 9c84cd0857df4b75ae959a38b8132eaf: Deleting consensus metadata
I20260620 00:16:27.819906 17935 catalog_manager.cc:4841] TS 9c84cd0857df4b75ae959a38b8132eaf (127.0.98.2:36781): tablet a84be1109649492381741d8285611bec (table test-table [id=3abeb059eccd4f29bde896eb69c97e47]) successfully deleted
I20260620 00:16:27.819913 17933 catalog_manager.cc:4841] TS e718d25062a84491acb91c9cd32be7f9 (127.0.98.3:35463): tablet a84be1109649492381741d8285611bec (table test-table [id=3abeb059eccd4f29bde896eb69c97e47]) successfully deleted
W20260620 00:16:27.820988 17934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40031 (error 108)
I20260620 00:16:27.821177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18140
I20260620 00:16:27.823041 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:27.822916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40031 (local address 127.0.0.1:50756)
0620 00:16:27.822961 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:27.823009 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 00:16:27.823197 17934 catalog_manager.cc:4584] TS b5877226937245198cac03f2d336977a (127.0.98.1:40031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a84be1109649492381741d8285611bec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40031: connect: Connection refused (error 111)
I20260620 00:16:27.827334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18271
I20260620 00:16:27.832482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 17918
2026-06-20T00:16:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (945 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:27 dist-test-slave-28bv krb5kdc[18441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:27 dist-test-slave-28bv krb5kdc[18441](info): set up 2 sockets
Jun 20 00:16:27 dist-test-slave-28bv krb5kdc[18441](info): commencing operation
krb5kdc: starting...
W20260620 00:16:28.900377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:28 dist-test-slave-28bv krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:29.027906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:29.110062 18454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:29.110296 18454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:29.110319 18454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:29.111475 18454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:29.111517 18454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:29.111531 18454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:29.111544 18454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:29.111557 18454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:29.113045 18454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:29.113247 18454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:29.113420 18454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:29.118482 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.118494 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.118480 18462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.118734 18454 server_base.cc:1034] running on GCE node
I20260620 00:16:29.119038 18454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:29.119361 18454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:29.120515 18454 hybrid_clock.cc:648] HybridClock initialized: now 1781914589120484 us; error 47 us; skew 500 ppm
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:29.127662 18454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:29.128234 18454 webserver.cc:460] Webserver started at http://127.0.98.62:39447/ using document root <none> and password file <none>
I20260620 00:16:29.128473 18454 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:29.128543 18454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:29.128741 18454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:29.129856 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "ff9723256ee74bb59208a81be2b6d2e1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "22d5175eef0c830b9a97fd660a3655c9"
server_key_iv: "c77d4852e17f08ce7b6e8edc18719d3b"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.130236 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff9723256ee74bb59208a81be2b6d2e1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "22d5175eef0c830b9a97fd660a3655c9"
server_key_iv: "c77d4852e17f08ce7b6e8edc18719d3b"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.131623 18454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:29.132464 18469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.132683 18454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:29.132763 18454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "ff9723256ee74bb59208a81be2b6d2e1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "22d5175eef0c830b9a97fd660a3655c9"
server_key_iv: "c77d4852e17f08ce7b6e8edc18719d3b"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.132835 18454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:29.151782 18454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:29.153949 18454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:29.154063 18454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:29.158406 18454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38583
I20260620 00:16:29.158643 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38583 every 8 connection(s)
I20260620 00:16:29.158843 18454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:29.159765 18522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:29.162014 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: Bootstrap starting.
I20260620 00:16:29.162563 18522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:29.162792 18522 log.cc:824] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:29.163048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18454
I20260620 00:16:29.163228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:29.163443 392 external_mini_cluster.cc:1428] Setting key 08ff3d74c526a921b0bdd74c201c7fe3
I20260620 00:16:29.163718 18522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: No bootstrap required, opened a new log
I20260620 00:16:29.165763 18522 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } }
I20260620 00:16:29.165954 18522 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:29.166005 18522 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff9723256ee74bb59208a81be2b6d2e1, State: Initialized, Role: FOLLOWER
I20260620 00:16:29.166139 18522 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } }
I20260620 00:16:29.166239 18522 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:29.166282 18522 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:29.166332 18522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:29.167068 18522 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } }
I20260620 00:16:29.167191 18522 leader_election.cc:302] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff9723256ee74bb59208a81be2b6d2e1; no voters:
I20260620 00:16:29.167407 18522 leader_election.cc:288] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:29.167426 18527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:29.167608 18527 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [term 1 LEADER]: Becoming Leader. State: Replica: ff9723256ee74bb59208a81be2b6d2e1, State: Running, Role: LEADER
I20260620 00:16:29.167685 18527 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } }
I20260620 00:16:29.167716 18522 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:29.168046 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff9723256ee74bb59208a81be2b6d2e1. Latest consensus state: current_term: 1 leader_uuid: "ff9723256ee74bb59208a81be2b6d2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } } }
I20260620 00:16:29.168056 18529 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff9723256ee74bb59208a81be2b6d2e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff9723256ee74bb59208a81be2b6d2e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38583 } } }
I20260620 00:16:29.168119 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:29.168164 18529 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:29.168356 18532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:29.168931 18532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:29.171159 18532 catalog_manager.cc:1347] Generated new cluster ID: f07b64d200324a2fb8bef12e9cbc1ade
I20260620 00:16:29.171224 18532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:29.176153 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.164001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.0.98.62:38583)
0620 00:16:29.164217 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.164228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.164262 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:16:29.165057 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.165065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.165115 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.165271 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:29.165926 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.167390 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.168474 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.169093 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.169199 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.172218 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:29.172222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:29.172223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:29.172236 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:29.174326 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.174766 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.174769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.174770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.174890 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.175106 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.175106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.175107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.175441 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:29.175591 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.175711 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.175873 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260620 00:16:29.181646 18532 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:29.182237 18532 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:29.186996 18532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: Generated new TSK 0
I20260620 00:16:29.187198 18532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:16:29.215907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:29.298906 18548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:29.299156 18548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:29.299199 18548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:29.300472 18548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:29.300571 18548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:29.300642 18548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:29.302228 18548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:29.302512 18548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:29.302762 18548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:29.308123 18556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.308144 18554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.308137 18553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.308169 18548 server_base.cc:1034] running on GCE node
I20260620 00:16:29.308647 18548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:29.308940 18548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:29.310086 18548 hybrid_clock.cc:648] HybridClock initialized: now 1781914589310068 us; error 41 us; skew 500 ppm
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:29.312815 18548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:16:29.313282 18548 webserver.cc:460] Webserver started at http://127.0.98.1:46167/ using document root <none> and password file <none>
I20260620 00:16:29.313514 18548 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:29.313557 18548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:29.313719 18548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:29.314893 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "f97ac920d696443e898f095b8898d0be"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f9a2018988fdcf756f5a537b9a632cf6"
server_key_iv: "66fbaf3aef753dea16081f15c654017c"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.315246 18548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f97ac920d696443e898f095b8898d0be"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f9a2018988fdcf756f5a537b9a632cf6"
server_key_iv: "66fbaf3aef753dea16081f15c654017c"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.316660 18548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:29.317476 18563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.317667 18548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:29.317806 18548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "f97ac920d696443e898f095b8898d0be"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f9a2018988fdcf756f5a537b9a632cf6"
server_key_iv: "66fbaf3aef753dea16081f15c654017c"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.317880 18548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:29.333884 18548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:29.336359 18548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:29.336530 18548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:29.336812 18548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:29.337121 18548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:29.337175 18548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.337241 18548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:29.337288 18548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.344039 18548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39157
I20260620 00:16:29.344082 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39157 every 8 connection(s)
I20260620 00:16:29.344412 18548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:29.351109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18548
I20260620 00:16:29.351233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:29.351465 392 external_mini_cluster.cc:1428] Setting key d3882ba3a2d7e55f45707951b04906dc
I20260620 00:16:29.354498 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.345227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43707 (local address 127.0.98.62:38583)
0620 00:16:29.345311 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.345313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.345520 (+ 207us) server_negotiation.cc:407] Connection header received
0620 00:16:29.346422 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.346423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.346433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.346504 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:29.347094 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.347604 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.348629 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.348841 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.348906 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.351148 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:29.351149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:29.351149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:29.351156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:29.353024 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.353456 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.353458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.353458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.353523 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.353826 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.353827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.353827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.353951 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:29.354032 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.354309 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.354407 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:16:29.354730 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.345163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38583 (local address 127.0.98.1:43707)
0620 00:16:29.345388 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.345431 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.346332 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.346563 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.346566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.346826 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:29.347019 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.347023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.347718 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.347719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.348514 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.348516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.348922 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.348923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.349035 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.349598 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:29.349607 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:29.351036 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:29.353102 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.353111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.353113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.353340 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.353590 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.353590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.353591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.353743 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.354022 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:29.354029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:29.354179 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.354289 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.354450 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
I20260620 00:16:29.355266 18677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38583
I20260620 00:16:29.355376 18677 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:29.355568 18677 heartbeater.cc:511] Master 127.0.98.62:38583 requested a full tablet report, sending...
I20260620 00:16:29.356177 18486 ts_manager.cc:194] Registered new tserver with Master: f97ac920d696443e898f095b8898d0be (127.0.98.1:39157)
I20260620 00:16:29.356874 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43707
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:16:29.379133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:29.469692 18682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:29.469947 18682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:29.469977 18682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:29.471527 18682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:29.471606 18682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:29.471642 18682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:29.474537 18682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:29.475018 18682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:29.475397 18682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:29.482935 18687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.482987 18682 server_base.cc:1034] running on GCE node
W20260620 00:16:29.483001 18690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.482936 18688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.483655 18682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:29.484062 18682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:29.485291 18682 hybrid_clock.cc:648] HybridClock initialized: now 1781914589485271 us; error 37 us; skew 500 ppm
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:29.488400 18682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:16:29.489038 18682 webserver.cc:460] Webserver started at http://127.0.98.2:44525/ using document root <none> and password file <none>
I20260620 00:16:29.489334 18682 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:29.489396 18682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:29.489548 18682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:29.490809 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff55072474c24ccca887f7be5f942ac1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "fe0ed5e557be48616ef3496b0a9eccea"
server_key_iv: "a63da7791ac7e7e889559dada46a0562"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.491341 18682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff55072474c24ccca887f7be5f942ac1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "fe0ed5e557be48616ef3496b0a9eccea"
server_key_iv: "a63da7791ac7e7e889559dada46a0562"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.493144 18682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 00:16:29.494207 18697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.494405 18682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:29.494504 18682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "ff55072474c24ccca887f7be5f942ac1"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "fe0ed5e557be48616ef3496b0a9eccea"
server_key_iv: "a63da7791ac7e7e889559dada46a0562"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.494601 18682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:29.506716 18682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:29.509712 18682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:29.509846 18682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:29.510054 18682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:29.510414 18682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:29.510457 18682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.510483 18682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:29.510495 18682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.517604 18682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40055
I20260620 00:16:29.517643 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40055 every 8 connection(s)
I20260620 00:16:29.518028 18682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:29.524871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18682
I20260620 00:16:29.525018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:29.525245 392 external_mini_cluster.cc:1428] Setting key d424ffcf7d94624b44d9634120b4e6c0
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:29.532037 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.518878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59711 (local address 127.0.98.62:38583)
0620 00:16:29.518967 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.518970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.519140 (+ 170us) server_negotiation.cc:407] Connection header received
0620 00:16:29.520220 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.520222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.520241 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.520363 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:29.521357 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.522105 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.523976 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.524257 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.524356 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.528121 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:29.528123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:29.528124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:29.528135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:29.529872 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.530777 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.530778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.530779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.530858 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.531240 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.531241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.531241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.531391 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:29.531475 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.531827 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.531931 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:16:29.532294 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.518816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38583 (local address 127.0.98.2:59711)
0620 00:16:29.519006 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.519047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.520009 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.520504 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.520508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.520985 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:29.521210 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.521215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.522354 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.522357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.523621 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.523627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.524432 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.524434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.524819 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.525588 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:29.525742 (+ 154us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:29.527977 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:29.530001 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.530014 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.530017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.530394 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.530940 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.530941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.530941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.531140 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.531475 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:29.531486 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:29.531665 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.531788 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.531998 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260620 00:16:29.532896 18811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38583
I20260620 00:16:29.533007 18811 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:29.533229 18811 heartbeater.cc:511] Master 127.0.98.62:38583 requested a full tablet report, sending...
I20260620 00:16:29.533746 18486 ts_manager.cc:194] Registered new tserver with Master: ff55072474c24ccca887f7be5f942ac1 (127.0.98.2:40055)
I20260620 00:16:29.534196 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59711
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:16:29.557191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:29.645156 18816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:29.645481 18816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:29.645531 18816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:29.646997 18816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:29.647133 18816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:29.647202 18816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:29.649608 18816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:29.650004 18816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:29.650291 18816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:29.656487 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:29.656490 18822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.656816 18816 server_base.cc:1034] running on GCE node
W20260620 00:16:29.656497 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:29.657368 18816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:29.657622 18816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:29.658874 18816 hybrid_clock.cc:648] HybridClock initialized: now 1781914589658857 us; error 32 us; skew 500 ppm
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:29.662463 18816 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:16:29.663045 18816 webserver.cc:460] Webserver started at http://127.0.98.3:37739/ using document root <none> and password file <none>
I20260620 00:16:29.663297 18816 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:29.663341 18816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:29.663443 18816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:29.664801 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "868a6062eb85412da515b811a98c13de"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f6d84e256613a90e759a1d63ce77d1f4"
server_key_iv: "0681d8252c9f66f22c042df83d673bd2"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.665220 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "868a6062eb85412da515b811a98c13de"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f6d84e256613a90e759a1d63ce77d1f4"
server_key_iv: "0681d8252c9f66f22c042df83d673bd2"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.666937 18816 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:16:29.668138 18831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.668524 18816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:29.668767 18816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "868a6062eb85412da515b811a98c13de"
format_stamp: "Formatted at 2026-06-20 00:16:29 on dist-test-slave-28bv"
server_key: "f6d84e256613a90e759a1d63ce77d1f4"
server_key_iv: "0681d8252c9f66f22c042df83d673bd2"
server_key_version: "encryptionkey@0"
I20260620 00:16:29.668926 18816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:29.681603 18816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:29.684749 18816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:29.684975 18816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:29.685271 18816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:29.685658 18816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:29.685729 18816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.685776 18816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:29.685808 18816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:29.693861 18816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42439
I20260620 00:16:29.693897 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42439 every 8 connection(s)
I20260620 00:16:29.694381 18816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:16:29 dist-test-slave-28bv krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:29.703948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18816
I20260620 00:16:29.704067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:29.704309 392 external_mini_cluster.cc:1428] Setting key dcf2640f4c3983245fb03749e45dfbde
I20260620 00:16:29.708715 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.695366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47929 (local address 127.0.98.62:38583)
0620 00:16:29.695456 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.695458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.695596 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:16:29.696897 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.696899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.696918 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.697075 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:29.697974 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.698617 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.700371 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.700728 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.700804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.703891 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:29.703894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:29.703894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:29.703907 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:29.705868 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.706911 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.706912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.706912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.706992 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.707569 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.707571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.707571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.707728 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:29.707825 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.708285 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.708614 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 00:16:29.708813 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.695247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38583 (local address 127.0.98.3:47929)
0620 00:16:29.695454 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.695496 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.696684 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.697195 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.697200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.697507 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:29.697829 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.697835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.698892 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.698895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.700223 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.700229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.700920 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.700922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.701093 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.701844 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:29.701855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:29.703648 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:29.705984 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.705995 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.705997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.706781 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.707313 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:29.707314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:29.707314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:29.707450 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:29.707837 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:29.707844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:29.708008 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.708131 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.708323 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260620 00:16:29.709547 18945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38583
I20260620 00:16:29.709745 18945 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:29.710016 18945 heartbeater.cc:511] Master 127.0.98.62:38583 requested a full tablet report, sending...
I20260620 00:16:29.710711 18486 ts_manager.cc:194] Registered new tserver with Master: 868a6062eb85412da515b811a98c13de (127.0.98.3:42439)
I20260620 00:16:29.711203 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47929
I20260620 00:16:29.716552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:29.727939 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.718258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.98.62:38583)
0620 00:16:29.718349 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.718351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.718364 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:16:29.718400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.718401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.718413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.718771 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:29.719495 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.720114 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.721217 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.721544 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.721649 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.723629 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:29.723631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:29.723632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:29.723640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:29.725670 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.726561 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.726563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.726564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.726662 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:29.727037 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:29.727039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:29.727039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:29.727267 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:29.727401 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.727765 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.727861 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:16:29.730867 18486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:29.733760 18486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:29.742944 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.738419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.2:40055)
0620 00:16:29.738623 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.738629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.738851 (+ 222us) server_negotiation.cc:407] Connection header received
0620 00:16:29.738901 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.738904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.738921 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.738997 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:29.739201 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.739987 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.741856 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742437 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742559 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742642 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.742802 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.742846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":128,"threads_started":1}
I20260620 00:16:29.742938 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.738250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40055 (local address 127.0.0.1:43654)
0620 00:16:29.738720 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.738731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.738786 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.739078 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.739079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.739080 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:29.739119 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.739120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.740146 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.740148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.741734 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.741736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742563 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.742665 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742683 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.742765 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.742841 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":98,"thread_start_us":84,"threads_started":1}
I20260620 00:16:29.743183 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.738105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.98.3:42439)
0620 00:16:29.738387 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.738391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.738402 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:16:29.738475 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.738478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.738498 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.738596 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:29.738967 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.740257 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.741816 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742562 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742703 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.743011 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.743066 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":94,"threads_started":1}
I20260620 00:16:29.743418 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.738047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.98.1:39157)
0620 00:16:29.738287 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.738293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.738814 (+ 521us) server_negotiation.cc:407] Connection header received
0620 00:16:29.738871 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.738874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.738892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.738967 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:29.739217 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.740385 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.741748 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742556 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742639 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742703 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.743213 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.743327 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
I20260620 00:16:29.743597 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.737953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39157 (local address 127.0.0.1:46186)
0620 00:16:29.738657 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.738670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.738740 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.739073 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.739074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.739076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:29.739129 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.739131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.740497 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.740498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.741640 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.741644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742751 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.742839 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742842 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.742894 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.743510 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":11,"thread_start_us":101,"threads_started":1}
I20260620 00:16:29.744100 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.737935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42439 (local address 127.0.0.1:37786)
0620 00:16:29.738170 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.738219 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.738304 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.738705 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.738709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.738713 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:29.738893 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.738895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.740381 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.740382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.741730 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.741734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.742749 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.742750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.742833 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.742837 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.742894 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.742959 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":16,"thread_start_us":87,"threads_started":1}
I20260620 00:16:29.744742 18745 tablet_service.cc:1467] Processing CreateTablet for tablet 425e567a69f545939eb1226576912ed5 (DEFAULT_TABLE table=test-table [id=399bea5de4dc4a4091d83f1c9a8845db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:29.744740 18611 tablet_service.cc:1467] Processing CreateTablet for tablet 425e567a69f545939eb1226576912ed5 (DEFAULT_TABLE table=test-table [id=399bea5de4dc4a4091d83f1c9a8845db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:29.745179 18745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425e567a69f545939eb1226576912ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:29.745184 18611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425e567a69f545939eb1226576912ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:29.745326 18879 tablet_service.cc:1467] Processing CreateTablet for tablet 425e567a69f545939eb1226576912ed5 (DEFAULT_TABLE table=test-table [id=399bea5de4dc4a4091d83f1c9a8845db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:29.745617 18879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 425e567a69f545939eb1226576912ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:29.749207 18964 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Bootstrap starting.
I20260620 00:16:29.749365 18963 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Bootstrap starting.
I20260620 00:16:29.749058 18962 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Bootstrap starting.
I20260620 00:16:29.750080 18964 tablet_bootstrap.cc:654] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:29.750361 18962 tablet_bootstrap.cc:654] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:29.750744 18964 log.cc:824] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:29.751127 18962 log.cc:824] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:29.751950 18964 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: No bootstrap required, opened a new log
I20260620 00:16:29.751967 18963 tablet_bootstrap.cc:654] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:29.751991 18962 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: No bootstrap required, opened a new log
I20260620 00:16:29.752041 18964 ts_tablet_manager.cc:1397] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:16:29.752205 18962 ts_tablet_manager.cc:1397] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:16:29.752600 18963 log.cc:824] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:29.753437 18963 tablet_bootstrap.cc:492] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: No bootstrap required, opened a new log
I20260620 00:16:29.753536 18963 ts_tablet_manager.cc:1397] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 00:16:29.754457 18964 raft_consensus.cc:348] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.754457 18962 raft_consensus.cc:348] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.754695 18962 raft_consensus.cc:374] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:29.754695 18964 raft_consensus.cc:374] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:29.754738 18962 raft_consensus.cc:729] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f97ac920d696443e898f095b8898d0be, State: Initialized, Role: FOLLOWER
I20260620 00:16:29.754738 18964 raft_consensus.cc:729] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868a6062eb85412da515b811a98c13de, State: Initialized, Role: FOLLOWER
I20260620 00:16:29.754909 18964 consensus_queue.cc:260] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.754909 18962 consensus_queue.cc:260] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.755288 18945 heartbeater.cc:503] Master 127.0.98.62:38583 was elected leader, sending a full tablet report...
I20260620 00:16:29.755321 18962 ts_tablet_manager.cc:1428] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:16:29.755594 18677 heartbeater.cc:503] Master 127.0.98.62:38583 was elected leader, sending a full tablet report...
I20260620 00:16:29.755266 18964 ts_tablet_manager.cc:1428] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:16:29.755987 18963 raft_consensus.cc:348] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.756186 18963 raft_consensus.cc:374] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:29.756234 18963 raft_consensus.cc:729] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff55072474c24ccca887f7be5f942ac1, State: Initialized, Role: FOLLOWER
I20260620 00:16:29.756371 18963 consensus_queue.cc:260] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.756762 18963 ts_tablet_manager.cc:1428] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260620 00:16:29.756762 18811 heartbeater.cc:503] Master 127.0.98.62:38583 was elected leader, sending a full tablet report...
W20260620 00:16:29.768934 18812 tablet.cc:2369] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:16:29.845278 18678 tablet.cc:2369] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:29.869566 18969 raft_consensus.cc:482] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:29.869727 18969 raft_consensus.cc:504] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.870191 18969 leader_election.cc:288] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 868a6062eb85412da515b811a98c13de (127.0.98.3:42439), ff55072474c24ccca887f7be5f942ac1 (127.0.98.2:40055)
I20260620 00:16:29.874388 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.870570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46741 (local address 127.0.98.2:40055)
0620 00:16:29.870634 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.870637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.870761 (+ 124us) server_negotiation.cc:407] Connection header received
0620 00:16:29.870981 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.871001 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.871019 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.871101 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:29.871431 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.872094 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.873456 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.873953 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.874040 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.874055 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.874249 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.874300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:16:29.874401 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.870636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39885 (local address 127.0.98.3:42439)
0620 00:16:29.870685 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.870688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.870757 (+ 69us) server_negotiation.cc:407] Connection header received
0620 00:16:29.870857 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.870858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.870870 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.871042 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:29.871223 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.871860 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.873289 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.873969 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.874072 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.874088 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.874280 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.874329 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:16:29.874413 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.870444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40055 (local address 127.0.98.1:46741)
0620 00:16:29.870659 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.870674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.870776 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.871233 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.871234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.871237 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:29.871320 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.871322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.872195 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.872196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.873295 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.873298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.874058 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.874060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.874152 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.874164 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.874212 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.874287 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260620 00:16:29.874416 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.870497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42439 (local address 127.0.98.1:39885)
0620 00:16:29.870661 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.870673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:29.870776 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:29.871046 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:29.871047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:29.871050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:29.871154 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.871157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.871946 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.871947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.873189 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:29.873193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.874086 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.874088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:29.874176 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.874180 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:29.874261 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:16:29.874309 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":22,"thread_start_us":83,"threads_started":1}
I20260620 00:16:29.874802 18765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425e567a69f545939eb1226576912ed5" candidate_uuid: "f97ac920d696443e898f095b8898d0be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff55072474c24ccca887f7be5f942ac1" is_pre_election: true
I20260620 00:16:29.875002 18765 raft_consensus.cc:2454] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97ac920d696443e898f095b8898d0be in term 0.
I20260620 00:16:29.875054 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425e567a69f545939eb1226576912ed5" candidate_uuid: "f97ac920d696443e898f095b8898d0be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868a6062eb85412da515b811a98c13de" is_pre_election: true
I20260620 00:16:29.875204 18899 raft_consensus.cc:2454] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f97ac920d696443e898f095b8898d0be in term 0.
I20260620 00:16:29.875315 18565 leader_election.cc:302] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f97ac920d696443e898f095b8898d0be, ff55072474c24ccca887f7be5f942ac1; no voters:
I20260620 00:16:29.875509 18969 raft_consensus.cc:2781] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:29.875588 18969 raft_consensus.cc:482] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:29.875617 18969 raft_consensus.cc:3037] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:29.876425 18969 raft_consensus.cc:504] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.876636 18969 leader_election.cc:288] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [CANDIDATE]: Term 1 election: Requested vote from peers 868a6062eb85412da515b811a98c13de (127.0.98.3:42439), ff55072474c24ccca887f7be5f942ac1 (127.0.98.2:40055)
I20260620 00:16:29.876926 18765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425e567a69f545939eb1226576912ed5" candidate_uuid: "f97ac920d696443e898f095b8898d0be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff55072474c24ccca887f7be5f942ac1"
I20260620 00:16:29.876914 18899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "425e567a69f545939eb1226576912ed5" candidate_uuid: "f97ac920d696443e898f095b8898d0be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868a6062eb85412da515b811a98c13de"
I20260620 00:16:29.877022 18899 raft_consensus.cc:3037] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:29.877022 18765 raft_consensus.cc:3037] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:29.878157 18765 raft_consensus.cc:2454] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97ac920d696443e898f095b8898d0be in term 1.
I20260620 00:16:29.878157 18899 raft_consensus.cc:2454] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f97ac920d696443e898f095b8898d0be in term 1.
I20260620 00:16:29.878422 18566 leader_election.cc:302] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868a6062eb85412da515b811a98c13de, f97ac920d696443e898f095b8898d0be; no voters:
I20260620 00:16:29.878604 18969 raft_consensus.cc:2781] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:29.878877 18969 raft_consensus.cc:686] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [term 1 LEADER]: Becoming Leader. State: Replica: f97ac920d696443e898f095b8898d0be, State: Running, Role: LEADER
I20260620 00:16:29.879176 18969 consensus_queue.cc:237] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } }
I20260620 00:16:29.880249 18485 catalog_manager.cc:5495] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be reported cstate change: term changed from 0 to 1, leader changed from <none> to f97ac920d696443e898f095b8898d0be (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f97ac920d696443e898f095b8898d0be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f97ac920d696443e898f095b8898d0be" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260620 00:16:29.895112 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.892287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.98.1:39157)
0620 00:16:29.892380 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:29.892382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:29.892391 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:29.892431 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:29.892432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:29.892454 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:29.892526 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:29.892774 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.893267 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.894062 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:29.894278 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:29.894348 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:29.894469 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:29.894565 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:29.894724 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:29.894993 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 00:16:29.895010 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 00:16:29.897297 18899 raft_consensus.cc:1261] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 1 FOLLOWER]: Refusing update from remote peer f97ac920d696443e898f095b8898d0be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:29.897398 18765 raft_consensus.cc:1261] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 1 FOLLOWER]: Refusing update from remote peer f97ac920d696443e898f095b8898d0be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:29.897779 18969 consensus_queue.cc:1035] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff55072474c24ccca887f7be5f942ac1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:29.897993 18973 consensus_queue.cc:1035] T 425e567a69f545939eb1226576912ed5 P f97ac920d696443e898f095b8898d0be [LEADER]: Connected to new peer: Peer: permanent_uuid: "868a6062eb85412da515b811a98c13de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:29.901419 18977 mvcc.cc:205] Tried to move back new op lower bound from 7298722160215478272 to 7298722160145727488. Current Snapshot: MvccSnapshot[applied={T|T < 7298722160215478272}]
I20260620 00:16:29.902575 18978 mvcc.cc:205] Tried to move back new op lower bound from 7298722160215478272 to 7298722160145727488. Current Snapshot: MvccSnapshot[applied={T|T < 7298722160215478272 or (T in {7298722160215478272})}]
I20260620 00:16:29.903888 18485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:29.904017 18485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:29.904923 18485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff9723256ee74bb59208a81be2b6d2e1: Sending DeleteTablet for 3 replicas of tablet 425e567a69f545939eb1226576912ed5
I20260620 00:16:29.905435 18879 tablet_service.cc:1514] Processing DeleteTablet for tablet 425e567a69f545939eb1226576912ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37786
I20260620 00:16:29.905557 18745 tablet_service.cc:1514] Processing DeleteTablet for tablet 425e567a69f545939eb1226576912ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43654
I20260620 00:16:29.905696 18985 tablet_replica.cc:330] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: stopping tablet replica
I20260620 00:16:29.905828 18985 raft_consensus.cc:2229] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:29.905858 18986 tablet_replica.cc:330] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: stopping tablet replica
I20260620 00:16:29.905946 18985 raft_consensus.cc:2258] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:29.905951 18986 raft_consensus.cc:2229] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:29.905942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18548
I20260620 00:16:29.908464 18985 ts_tablet_manager.cc:1905] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:29.906061 18986 raft_consensus.cc:2258] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:29.909227 18986 ts_tablet_manager.cc:1905] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:29.910725 18985 ts_tablet_manager.cc:1918] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:29.910768 18986 ts_tablet_manager.cc:1918] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:29.910822 18985 log.cc:1196] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/425e567a69f545939eb1226576912ed5
I20260620 00:16:29.910881 18986 log.cc:1196] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/425e567a69f545939eb1226576912ed5
I20260620 00:16:29.911139 18985 ts_tablet_manager.cc:1939] T 425e567a69f545939eb1226576912ed5 P 868a6062eb85412da515b811a98c13de: Deleting consensus metadata
I20260620 00:16:29.911239 18986 ts_tablet_manager.cc:1939] T 425e567a69f545939eb1226576912ed5 P ff55072474c24ccca887f7be5f942ac1: Deleting consensus metadata
I20260620 00:16:29.911742 18472 catalog_manager.cc:4841] TS 868a6062eb85412da515b811a98c13de (127.0.98.3:42439): tablet 425e567a69f545939eb1226576912ed5 (table test-table [id=399bea5de4dc4a4091d83f1c9a8845db]) successfully deleted
I20260620 00:16:29.911798 18471 catalog_manager.cc:4841] TS ff55072474c24ccca887f7be5f942ac1 (127.0.98.2:40055): tablet 425e567a69f545939eb1226576912ed5 (table test-table [id=399bea5de4dc4a4091d83f1c9a8845db]) successfully deleted
W20260620 00:16:29.912323 18471 connection.cc:535] client connection to 127.0.98.1:39157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:16:29.912433 18471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:16:29.912781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18682
I20260620 00:16:29.913508 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:29.912952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39157 (local address 127.0.0.1:46206)
0620 00:16:29.913401 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:29.913459 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
W20260620 00:16:29.913758 18471 catalog_manager.cc:4584] TS f97ac920d696443e898f095b8898d0be (127.0.98.1:39157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 425e567a69f545939eb1226576912ed5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39157: connect: Connection refused (error 111)
I20260620 00:16:29.919279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18816
I20260620 00:16:29.924996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18454
2026-06-20T00:16:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2093 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T00:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:29Z Disabled control of system clock
I20260620 00:16:29.955933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37837
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:30.054965 18991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:30.055265 18991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:30.055294 18991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:30.056845 18991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:30.056914 18991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:30.056929 18991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:30.056944 18991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:30.056958 18991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:30.058666 18991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:30.059027 18991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:30.059415 18991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:30.065999 18997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.065974 18999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.065974 18996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.067093 18991 server_base.cc:1034] running on GCE node
I20260620 00:16:30.067807 18991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:30.068254 18991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:30.069509 18991 hybrid_clock.cc:648] HybridClock initialized: now 1781914590069443 us; error 88 us; skew 500 ppm
I20260620 00:16:30.072361 18991 webserver.cc:460] Webserver started at http://127.0.98.62:43219/ using document root <none> and password file <none>
I20260620 00:16:30.072826 18991 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:30.072917 18991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:30.073107 18991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:30.074532 18991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "1c10d7508da349398132c9d4fa43fcb6"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "5728b7c35f6cd943d3067b93e54998f3"
server_key_iv: "74a3b7b3f95829926387fb139ba999cf"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.074997 18991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c10d7508da349398132c9d4fa43fcb6"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "5728b7c35f6cd943d3067b93e54998f3"
server_key_iv: "74a3b7b3f95829926387fb139ba999cf"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.076771 18991 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 00:16:30.077898 19005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.078359 18991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:30.078529 18991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "1c10d7508da349398132c9d4fa43fcb6"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "5728b7c35f6cd943d3067b93e54998f3"
server_key_iv: "74a3b7b3f95829926387fb139ba999cf"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.078630 18991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:30.094952 18991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:30.095343 18991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:30.095510 18991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:30.100288 18991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37837
I20260620 00:16:30.100324 19057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37837 every 8 connection(s)
I20260620 00:16:30.101068 18991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:30.102456 19058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:30.102811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 18991
I20260620 00:16:30.102918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:30.103150 392 external_mini_cluster.cc:1428] Setting key 7d029de97546f369f92c51b9cf63b2d9
I20260620 00:16:30.106316 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: Bootstrap starting.
I20260620 00:16:30.107206 19058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:30.107584 19058 log.cc:824] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:30.108492 19058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: No bootstrap required, opened a new log
I20260620 00:16:30.111038 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.104165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.0.98.62:37837)
0620 00:16:30.104396 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.104407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.104443 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:16:30.105050 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.105061 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.105213 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.105400 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:30.105731 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.107172 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.108215 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.108906 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.109051 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.109887 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:30.109893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:30.109893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:30.110130 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:30.110190 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:30.110209 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:30.110291 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.110642 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.110774 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 00:16:30.111620 19058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260620 00:16:30.111836 19058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:30.111872 19058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c10d7508da349398132c9d4fa43fcb6, State: Initialized, Role: FOLLOWER
I20260620 00:16:30.112006 19058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260620 00:16:30.112102 19058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:30.112167 19058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:30.112226 19058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:30.113025 19058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260620 00:16:30.113137 19058 leader_election.cc:302] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c10d7508da349398132c9d4fa43fcb6; no voters:
I20260620 00:16:30.113416 19058 leader_election.cc:288] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:30.113524 19063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:30.113798 19063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [term 1 LEADER]: Becoming Leader. State: Replica: 1c10d7508da349398132c9d4fa43fcb6, State: Running, Role: LEADER
I20260620 00:16:30.113796 19058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:30.113925 19063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } }
I20260620 00:16:30.114369 19064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c10d7508da349398132c9d4fa43fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } } }
I20260620 00:16:30.114372 19065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c10d7508da349398132c9d4fa43fcb6. Latest consensus state: current_term: 1 leader_uuid: "1c10d7508da349398132c9d4fa43fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c10d7508da349398132c9d4fa43fcb6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37837 } } }
I20260620 00:16:30.114496 19064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:30.114523 19065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:30.115037 19072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:30.115576 19072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:30.117409 19072 catalog_manager.cc:1347] Generated new cluster ID: 991e07747f1941729555dbab4904d3ac
I20260620 00:16:30.117480 19072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:30.124385 19072 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:30.125067 19072 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:30.134194 19072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: Generated new TSK 0
I20260620 00:16:30.134445 19072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:30.142053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:30.239387 19082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:30.239645 19082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:30.239670 19082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:30.241117 19082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:30.241183 19082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:30.241214 19082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:30.243351 19082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:30.243708 19082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:30.244088 19082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:30.251329 19087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.251343 19088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.251636 19082 server_base.cc:1034] running on GCE node
W20260620 00:16:30.251343 19090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.251969 19082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:30.252297 19082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:30.253502 19082 hybrid_clock.cc:648] HybridClock initialized: now 1781914590253482 us; error 32 us; skew 500 ppm
I20260620 00:16:30.255076 19082 webserver.cc:460] Webserver started at http://127.0.98.1:41981/ using document root <none> and password file <none>
I20260620 00:16:30.255363 19082 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:30.255435 19082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:30.255605 19082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:30.256840 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "35c3fd31daf941aeaa47117a163483ff"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "47507593e356e5bd9dca722408b6a0ef"
server_key_iv: "656bf3f321691d0d6593835b791f3941"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.257242 19082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35c3fd31daf941aeaa47117a163483ff"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "47507593e356e5bd9dca722408b6a0ef"
server_key_iv: "656bf3f321691d0d6593835b791f3941"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.258746 19082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:16:30.259598 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.259821 19082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:30.259931 19082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "35c3fd31daf941aeaa47117a163483ff"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "47507593e356e5bd9dca722408b6a0ef"
server_key_iv: "656bf3f321691d0d6593835b791f3941"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.260015 19082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:30.294050 19082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:30.294458 19082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:30.294632 19082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:30.294912 19082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:30.295322 19082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:30.295383 19082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.295428 19082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:30.295459 19082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.303692 19082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42411
I20260620 00:16:30.303735 19209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42411 every 8 connection(s)
I20260620 00:16:30.304160 19082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:30.309240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19082
I20260620 00:16:30.309361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:30.309713 392 external_mini_cluster.cc:1428] Setting key 6d7a5fb9c97ccf97b7e0580e229c8ac5
I20260620 00:16:30.311358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:16:30.311825 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.305213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47607 (local address 127.0.98.62:37837)
0620 00:16:30.305293 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.305295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.305551 (+ 256us) server_negotiation.cc:407] Connection header received
0620 00:16:30.306186 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.306189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.306217 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.306324 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:30.307226 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.307848 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.309229 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.309774 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.309868 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.311279 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:30.311282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:30.311282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:30.311290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:30.311315 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:30.311322 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:30.311441 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.311666 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.311717 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:16:30.312145 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.305176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.1:47607)
0620 00:16:30.305412 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.305454 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.305979 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.306450 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.306454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.306464 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:30.307043 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.307048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.308002 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.308003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.309100 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.309103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.309882 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.309883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.310062 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.311008 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:30.311009 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:30.311066 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:30.311113 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:30.311432 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:30.311450 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:30.311456 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.311540 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.311733 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 00:16:30.313024 19210 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260620 00:16:30.313191 19210 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:30.313654 19210 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260620 00:16:30.314700 19022 ts_manager.cc:194] Registered new tserver with Master: 35c3fd31daf941aeaa47117a163483ff (127.0.98.1:42411)
I20260620 00:16:30.315608 19022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47607
W20260620 00:16:30.401539 19213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:30.401824 19213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:30.401862 19213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:30.403103 19213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:30.403182 19213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:30.403226 19213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:30.404765 19213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:30.405066 19213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:30.405400 19213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:30.410909 19218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.410979 19221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.411132 19213 server_base.cc:1034] running on GCE node
W20260620 00:16:30.411041 19219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.411446 19213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:30.411772 19213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:30.412972 19213 hybrid_clock.cc:648] HybridClock initialized: now 1781914590412939 us; error 56 us; skew 500 ppm
I20260620 00:16:30.414652 19213 webserver.cc:460] Webserver started at http://127.0.98.2:34143/ using document root <none> and password file <none>
I20260620 00:16:30.414953 19213 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:30.415032 19213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:30.415211 19213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:30.416350 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "8773d3719f088411071ab8544d64a8e0"
server_key_iv: "06f71e3e5446ec8dddc735e202f8e6e6"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.416759 19213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "8773d3719f088411071ab8544d64a8e0"
server_key_iv: "06f71e3e5446ec8dddc735e202f8e6e6"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.418223 19213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:30.419157 19227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.419459 19213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:30.419646 19213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "8773d3719f088411071ab8544d64a8e0"
server_key_iv: "06f71e3e5446ec8dddc735e202f8e6e6"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.419749 19213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:30.430637 19213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:30.431085 19213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:30.431217 19213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:30.431429 19213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:30.431785 19213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:30.431821 19213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.431847 19213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:30.431861 19213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.439846 19213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34747
I20260620 00:16:30.439958 19340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34747 every 8 connection(s)
I20260620 00:16:30.440505 19213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:30.447073 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.441713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40017 (local address 127.0.98.62:37837)
0620 00:16:30.441797 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.441799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.442059 (+ 260us) server_negotiation.cc:407] Connection header received
0620 00:16:30.442616 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.442617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.442630 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.442707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:30.443720 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.444234 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.445446 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.445696 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.445761 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.446723 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:30.446724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:30.446725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:30.446729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:30.446745 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:30.446750 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:30.446807 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.446961 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.447000 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:16:30.447299 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.441677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.2:40017)
0620 00:16:30.441900 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.441945 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.442473 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.442846 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.442850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.442860 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:30.443542 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.443549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.444368 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.444369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.445341 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.445345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.445791 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.445792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.445921 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.446496 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:30.446496 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:30.446538 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:30.446566 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:30.446842 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:30.446849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:30.446852 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.446925 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.447029 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260620 00:16:30.447911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19213
I20260620 00:16:30.447948 19341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260620 00:16:30.448045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:30.448053 19341 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:30.448271 392 external_mini_cluster.cc:1428] Setting key ad59f95bb522ae3b2d30927e674e82ca
I20260620 00:16:30.448285 19341 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260620 00:16:30.448769 19021 ts_manager.cc:194] Registered new tserver with Master: aa1207f62d3848d88fe5d5e38f5c3ed2 (127.0.98.2:34747)
I20260620 00:16:30.449188 19021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40017
I20260620 00:16:30.449334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:30.540288 19344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:30.540583 19344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:30.540612 19344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:30.542022 19344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:30.542088 19344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:30.542125 19344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:30.544130 19344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:30.544479 19344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:30.544957 19344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:30.551258 19352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.551262 19350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:30.551347 19349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:30.551574 19344 server_base.cc:1034] running on GCE node
I20260620 00:16:30.551815 19344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:30.552112 19344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:30.553311 19344 hybrid_clock.cc:648] HybridClock initialized: now 1781914590553288 us; error 60 us; skew 500 ppm
I20260620 00:16:30.554931 19344 webserver.cc:460] Webserver started at http://127.0.98.3:39149/ using document root <none> and password file <none>
I20260620 00:16:30.555202 19344 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:30.555284 19344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:30.555446 19344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:30.556640 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "c93ed41fcd314a52a7b28ec6e9b5759c"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "f10245c945db7fd72557c457cb9624b7"
server_key_iv: "bb68e1b68212eb469e362ed374006002"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.557037 19344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c93ed41fcd314a52a7b28ec6e9b5759c"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "f10245c945db7fd72557c457cb9624b7"
server_key_iv: "bb68e1b68212eb469e362ed374006002"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.558614 19344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:16:30.559537 19358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.559736 19344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:30.559859 19344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "c93ed41fcd314a52a7b28ec6e9b5759c"
format_stamp: "Formatted at 2026-06-20 00:16:30 on dist-test-slave-28bv"
server_key: "f10245c945db7fd72557c457cb9624b7"
server_key_iv: "bb68e1b68212eb469e362ed374006002"
server_key_version: "encryptionkey@0"
I20260620 00:16:30.559942 19344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:30.590617 19344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:30.590997 19344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:30.591181 19344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:30.591444 19344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:30.591799 19344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:30.591854 19344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.591892 19344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:30.591940 19344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:30.598559 19344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39859
I20260620 00:16:30.598600 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39859 every 8 connection(s)
I20260620 00:16:30.598985 19344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:30.605765 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.599815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44043 (local address 127.0.98.62:37837)
0620 00:16:30.599899 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.599901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.600155 (+ 254us) server_negotiation.cc:407] Connection header received
0620 00:16:30.601021 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.601022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.601043 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.601094 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:30.601886 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.602416 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.603620 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.603816 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.603886 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.605311 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:30.605313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:30.605313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:30.605320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:30.605336 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:30.605340 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:30.605395 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.605624 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.605665 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:16:30.605895 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.599814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37837 (local address 127.0.98.3:44043)
0620 00:16:30.600000 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.600038 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.600752 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.601202 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.601206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.601215 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:30.601770 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.601775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.602556 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.602556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.603486 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.603489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.603948 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.603949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.604102 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.605091 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:30.605092 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:30.605129 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:30.605163 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:30.605439 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:30.605446 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:30.605449 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.605569 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.605678 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260620 00:16:30.606112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19344
I20260620 00:16:30.606222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:30.606438 392 external_mini_cluster.cc:1428] Setting key db286fe36ff155fd0f7dee7de1bc0e9d
I20260620 00:16:30.606702 19472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37837
I20260620 00:16:30.606837 19472 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:30.607108 19472 heartbeater.cc:511] Master 127.0.98.62:37837 requested a full tablet report, sending...
I20260620 00:16:30.607548 19021 ts_manager.cc:194] Registered new tserver with Master: c93ed41fcd314a52a7b28ec6e9b5759c (127.0.98.3:39859)
I20260620 00:16:30.608048 19021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44043
I20260620 00:16:30.618912 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:30.623850 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.620613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.0.98.62:37837)
0620 00:16:30.620690 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.620692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.620703 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:16:30.620776 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.620777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.620788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.620844 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:30.621009 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.621520 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.622336 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.622548 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.622628 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.623557 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:30.623559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:30.623559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:30.623564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:30.623579 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:30.623583 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:30.623680 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.623773 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.623786 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:16:30.626549 19021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:30.628311 19021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:30.635356 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.631349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.0.98.1:42411)
0620 00:16:30.631605 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.631610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.631619 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:30.631656 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.631658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.631673 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.631828 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:30.632020 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.633155 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.634307 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.634924 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635013 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635116 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.635221 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.635260 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
I20260620 00:16:30.635489 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.630911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34747 (local address 127.0.0.1:59430)
0620 00:16:30.631299 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.631320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.631398 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.631624 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.631628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.631631 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:30.631719 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.631722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.633157 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.633158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.634233 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.634235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.635280 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.635334 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635336 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.635380 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.635418 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":69,"threads_started":1}
I20260620 00:16:30.635689 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.631109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.0.98.3:39859)
0620 00:16:30.631535 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.631539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.631548 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:30.631587 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.631589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.631602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.631696 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:30.631851 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.632768 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.634355 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.635092 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635417 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635506 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.635540 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.635594 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":107,"threads_started":1}
I20260620 00:16:30.635727 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.630900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39859 (local address 127.0.0.1:48652)
0620 00:16:30.631253 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.631295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.631365 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.631701 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.631702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.631704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:30.631769 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.631771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.632886 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.632887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.634268 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.634270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.635194 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.635258 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635262 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.635294 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.635335 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":86,"threads_started":1}
I20260620 00:16:30.635345 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.631151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.0.1:37354)
0620 00:16:30.631485 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.631495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.631535 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.631893 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.631894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.631896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:30.631944 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.631945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.633238 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.633239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.634221 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.634223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.635025 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.635113 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635132 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.635206 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.635261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":48,"threads_started":1}
I20260620 00:16:30.635984 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.630990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.98.2:34747)
0620 00:16:30.631240 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.631245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.631411 (+ 166us) server_negotiation.cc:407] Connection header received
0620 00:16:30.631508 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.631511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.631529 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.631607 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:30.631813 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.633064 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.634340 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.635090 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.635164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.635250 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.635730 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.635776 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260620 00:16:30.636669 19144 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 (DEFAULT_TABLE table=test-table [id=a3b512cf3eef4db49a8c7b9e49556dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:30.636832 19406 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 (DEFAULT_TABLE table=test-table [id=a3b512cf3eef4db49a8c7b9e49556dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:30.636795 19275 tablet_service.cc:1467] Processing CreateTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 (DEFAULT_TABLE table=test-table [id=a3b512cf3eef4db49a8c7b9e49556dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:30.637125 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:30.637159 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:30.637158 19406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:30.640269 19486 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Bootstrap starting.
I20260620 00:16:30.640780 19488 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Bootstrap starting.
I20260620 00:16:30.641219 19486 tablet_bootstrap.cc:654] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:30.641451 19488 tablet_bootstrap.cc:654] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:30.641551 19486 log.cc:824] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:30.641781 19488 log.cc:824] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:30.641980 19487 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Bootstrap starting.
I20260620 00:16:30.642477 19488 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: No bootstrap required, opened a new log
I20260620 00:16:30.642529 19486 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: No bootstrap required, opened a new log
I20260620 00:16:30.642591 19486 ts_tablet_manager.cc:1397] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:16:30.642581 19488 ts_tablet_manager.cc:1397] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:16:30.642634 19487 tablet_bootstrap.cc:654] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:30.642968 19487 log.cc:824] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:30.644253 19487 tablet_bootstrap.cc:492] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: No bootstrap required, opened a new log
I20260620 00:16:30.644333 19487 ts_tablet_manager.cc:1397] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:16:30.645169 19488 raft_consensus.cc:348] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.645365 19488 raft_consensus.cc:374] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:30.645411 19488 raft_consensus.cc:729] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa1207f62d3848d88fe5d5e38f5c3ed2, State: Initialized, Role: FOLLOWER
I20260620 00:16:30.645572 19488 consensus_queue.cc:260] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.645670 19486 raft_consensus.cc:348] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.645855 19486 raft_consensus.cc:374] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:30.645892 19488 ts_tablet_manager.cc:1428] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:16:30.645903 19486 raft_consensus.cc:729] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c3fd31daf941aeaa47117a163483ff, State: Initialized, Role: FOLLOWER
I20260620 00:16:30.646046 19341 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
I20260620 00:16:30.646070 19486 consensus_queue.cc:260] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.646351 19486 ts_tablet_manager.cc:1428] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:16:30.646413 19210 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
I20260620 00:16:30.647461 19487 raft_consensus.cc:348] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.647660 19487 raft_consensus.cc:374] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:30.647687 19487 raft_consensus.cc:729] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93ed41fcd314a52a7b28ec6e9b5759c, State: Initialized, Role: FOLLOWER
I20260620 00:16:30.647805 19487 consensus_queue.cc:260] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.648134 19487 ts_tablet_manager.cc:1428] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:16:30.648432 19472 heartbeater.cc:503] Master 127.0.98.62:37837 was elected leader, sending a full tablet report...
W20260620 00:16:30.691413 19342 tablet.cc:2369] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:30.799676 19494 raft_consensus.cc:482] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:30.799813 19494 raft_consensus.cc:504] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.800145 19494 leader_election.cc:288] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa1207f62d3848d88fe5d5e38f5c3ed2 (127.0.98.2:34747), 35c3fd31daf941aeaa47117a163483ff (127.0.98.1:42411)
I20260620 00:16:30.804399 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.800472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35643 (local address 127.0.98.2:34747)
0620 00:16:30.800554 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.800557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.800750 (+ 193us) server_negotiation.cc:407] Connection header received
0620 00:16:30.800779 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.800780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.800792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.800850 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:30.801005 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.801752 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.803366 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.803937 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.804020 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.804031 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.804279 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.804322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:16:30.804399 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.800478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46021 (local address 127.0.98.1:42411)
0620 00:16:30.800575 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.800577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.800658 (+ 81us) server_negotiation.cc:407] Connection header received
0620 00:16:30.800685 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.800686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.800695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.800779 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:30.801005 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.801762 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.803366 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.803933 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.804020 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.804031 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.804279 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.804322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:16:30.804418 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.800385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.98.3:46021)
0620 00:16:30.800448 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.800462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.800513 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.800803 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.800804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.800806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:30.800903 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.800906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.801880 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.801882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.803217 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.803223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.804084 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.804085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.804180 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.804184 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.804258 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.804314 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":39}
I20260620 00:16:30.804425 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.800385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34747 (local address 127.0.98.3:35643)
0620 00:16:30.800671 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.800685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:30.800727 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:30.800868 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:30.800869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:30.800870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:30.800928 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.800929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.801880 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.801882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.803224 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:30.803225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.804084 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.804085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:30.804180 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.804184 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:30.804253 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:16:30.804313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":111,"thread_start_us":220,"threads_started":1}
I20260620 00:16:30.804945 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4c9f7ee5d4aef9729cedf7ae25a05" candidate_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" is_pre_election: true
W20260620 00:16:30.805001 19211 tablet.cc:2369] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:30.804926 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4c9f7ee5d4aef9729cedf7ae25a05" candidate_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c3fd31daf941aeaa47117a163483ff" is_pre_election: true
I20260620 00:16:30.805132 19164 raft_consensus.cc:2454] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93ed41fcd314a52a7b28ec6e9b5759c in term 0.
I20260620 00:16:30.805132 19295 raft_consensus.cc:2454] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c93ed41fcd314a52a7b28ec6e9b5759c in term 0.
I20260620 00:16:30.805384 19362 leader_election.cc:302] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa1207f62d3848d88fe5d5e38f5c3ed2, c93ed41fcd314a52a7b28ec6e9b5759c; no voters:
I20260620 00:16:30.805531 19494 raft_consensus.cc:2781] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:30.805589 19494 raft_consensus.cc:482] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:30.805613 19494 raft_consensus.cc:3037] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:30.806315 19494 raft_consensus.cc:504] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.806470 19494 leader_election.cc:288] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [CANDIDATE]: Term 1 election: Requested vote from peers aa1207f62d3848d88fe5d5e38f5c3ed2 (127.0.98.2:34747), 35c3fd31daf941aeaa47117a163483ff (127.0.98.1:42411)
I20260620 00:16:30.806648 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4c9f7ee5d4aef9729cedf7ae25a05" candidate_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35c3fd31daf941aeaa47117a163483ff"
I20260620 00:16:30.806622 19295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aa4c9f7ee5d4aef9729cedf7ae25a05" candidate_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2"
I20260620 00:16:30.806727 19164 raft_consensus.cc:3037] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:30.806772 19295 raft_consensus.cc:3037] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:30.807484 19164 raft_consensus.cc:2454] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93ed41fcd314a52a7b28ec6e9b5759c in term 1.
I20260620 00:16:30.807679 19359 leader_election.cc:302] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c3fd31daf941aeaa47117a163483ff, c93ed41fcd314a52a7b28ec6e9b5759c; no voters:
I20260620 00:16:30.807690 19295 raft_consensus.cc:2454] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93ed41fcd314a52a7b28ec6e9b5759c in term 1.
I20260620 00:16:30.807801 19494 raft_consensus.cc:2781] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:30.808024 19494 raft_consensus.cc:686] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 1 LEADER]: Becoming Leader. State: Replica: c93ed41fcd314a52a7b28ec6e9b5759c, State: Running, Role: LEADER
I20260620 00:16:30.808133 19494 consensus_queue.cc:237] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } }
I20260620 00:16:30.809144 19022 catalog_manager.cc:5495] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c reported cstate change: term changed from 0 to 1, leader changed from <none> to c93ed41fcd314a52a7b28ec6e9b5759c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c93ed41fcd314a52a7b28ec6e9b5759c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 } health_report { overall_health: UNKNOWN } } }
I20260620 00:16:30.824342 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.821662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.98.3:39859)
0620 00:16:30.821730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:30.821732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:30.821742 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:30.821786 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:30.821786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:30.821807 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:30.821888 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:30.822022 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.822495 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.823332 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:30.823614 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:30.823690 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:30.823877 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:30.823967 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:30.824096 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:30.824192 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 00:16:30.824200 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:16:30.826300 19295 raft_consensus.cc:1261] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 1 FOLLOWER]: Refusing update from remote peer c93ed41fcd314a52a7b28ec6e9b5759c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:30.826309 19164 raft_consensus.cc:1261] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 1 FOLLOWER]: Refusing update from remote peer c93ed41fcd314a52a7b28ec6e9b5759c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:30.826735 19494 consensus_queue.cc:1035] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa1207f62d3848d88fe5d5e38f5c3ed2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:30.826894 19496 consensus_queue.cc:1035] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c3fd31daf941aeaa47117a163483ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:30.832943 19021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:30.833089 19021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:30.833963 19021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c10d7508da349398132c9d4fa43fcb6: Sending DeleteTablet for 3 replicas of tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05
I20260620 00:16:30.834360 19406 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:30 UTC) from {username='slave'} at 127.0.0.1:48652
I20260620 00:16:30.834443 19144 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:30 UTC) from {username='slave'} at 127.0.0.1:37354
I20260620 00:16:30.834539 19275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:30 UTC) from {username='slave'} at 127.0.0.1:59430
I20260620 00:16:30.834777 19510 tablet_replica.cc:330] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff: stopping tablet replica
I20260620 00:16:30.834863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19082
I20260620 00:16:30.834899 19510 raft_consensus.cc:2229] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:30.835019 19510 raft_consensus.cc:2258] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P 35c3fd31daf941aeaa47117a163483ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:30.835459 19511 tablet_replica.cc:330] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: stopping tablet replica
I20260620 00:16:30.835598 19511 raft_consensus.cc:2229] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:30.835711 19511 raft_consensus.cc:2258] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:30.835968 19509 tablet_replica.cc:330] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: stopping tablet replica
I20260620 00:16:30.836076 19509 raft_consensus.cc:2229] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:16:30.836216 19509 raft_consensus.cc:2258] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:30.837340 19511 ts_tablet_manager.cc:1905] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:30.838465 19509 ts_tablet_manager.cc:1905] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:30.839433 19511 ts_tablet_manager.cc:1918] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:30.839509 19511 log.cc:1196] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/9aa4c9f7ee5d4aef9729cedf7ae25a05
I20260620 00:16:30.839783 19511 ts_tablet_manager.cc:1939] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P aa1207f62d3848d88fe5d5e38f5c3ed2: Deleting consensus metadata
I20260620 00:16:30.840144 19509 ts_tablet_manager.cc:1918] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:30.840210 19509 log.cc:1196] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/9aa4c9f7ee5d4aef9729cedf7ae25a05
I20260620 00:16:30.840457 19509 ts_tablet_manager.cc:1939] T 9aa4c9f7ee5d4aef9729cedf7ae25a05 P c93ed41fcd314a52a7b28ec6e9b5759c: Deleting consensus metadata
I20260620 00:16:30.840953 19007 catalog_manager.cc:4841] TS c93ed41fcd314a52a7b28ec6e9b5759c (127.0.98.3:39859): tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 (table test-table [id=a3b512cf3eef4db49a8c7b9e49556dc8]) successfully deleted
W20260620 00:16:30.841689 19006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42411 (error 108)
I20260620 00:16:30.842026 19009 catalog_manager.cc:4841] TS aa1207f62d3848d88fe5d5e38f5c3ed2 (127.0.98.2:34747): tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05 (table test-table [id=a3b512cf3eef4db49a8c7b9e49556dc8]) successfully deleted
I20260620 00:16:30.842343 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:30.842175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42411 (local address 127.0.0.1:37358)
0620 00:16:30.842243 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:30.842310 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:16:30.842518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19213
W20260620 00:16:30.842558 19006 catalog_manager.cc:4584] TS 35c3fd31daf941aeaa47117a163483ff (127.0.98.1:42411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aa4c9f7ee5d4aef9729cedf7ae25a05: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42411: connect: Connection refused (error 111)
I20260620 00:16:30.848732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19344
I20260620 00:16:30.854904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 18991
2026-06-20T00:16:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:30 dist-test-slave-28bv krb5kdc[19515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:30 dist-test-slave-28bv krb5kdc[19515](info): set up 2 sockets
Jun 20 00:16:30 dist-test-slave-28bv krb5kdc[19515](info): commencing operation
krb5kdc: starting...
W20260620 00:16:31.936455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:32.072160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:32.159799 19528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:32.160080 19528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:32.160125 19528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:32.161448 19528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:32.161545 19528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:32.161571 19528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:32.161593 19528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:32.161623 19528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:32.163224 19528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:32.163549 19528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:32.163798 19528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:32.169488 19533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:32.169474 19536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:32.169473 19534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.169953 19528 server_base.cc:1034] running on GCE node
I20260620 00:16:32.170226 19528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:32.170578 19528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:32.171758 19528 hybrid_clock.cc:648] HybridClock initialized: now 1781914592171726 us; error 67 us; skew 500 ppm
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:32.175212 19528 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:32.175813 19528 webserver.cc:460] Webserver started at http://127.0.98.62:41161/ using document root <none> and password file <none>
I20260620 00:16:32.176123 19528 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:32.176185 19528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:32.176324 19528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:32.177672 19528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "280cee93be0a4294bb002af078caa4df"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "84bacdc6db6599907ecff01c8bc7a3bf"
server_key_iv: "36bad16099dfadc85fd620133e41aa00"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.178079 19528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "280cee93be0a4294bb002af078caa4df"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "84bacdc6db6599907ecff01c8bc7a3bf"
server_key_iv: "36bad16099dfadc85fd620133e41aa00"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.179651 19528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:32.180624 19543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.180795 19528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:32.180896 19528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "280cee93be0a4294bb002af078caa4df"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "84bacdc6db6599907ecff01c8bc7a3bf"
server_key_iv: "36bad16099dfadc85fd620133e41aa00"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.180991 19528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:32.191982 19528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:32.194603 19528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:32.194824 19528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:32.199599 19528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43087
I20260620 00:16:32.199645 19595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43087 every 8 connection(s)
I20260620 00:16:32.200062 19528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:32.201814 19596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:32.204228 19596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: Bootstrap starting.
I20260620 00:16:32.205309 19596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:32.205615 19596 log.cc:824] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:32.206544 19596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: No bootstrap required, opened a new log
I20260620 00:16:32.208045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19528
I20260620 00:16:32.208294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:32.208266 19596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } }
I20260620 00:16:32.208503 392 external_mini_cluster.cc:1428] Setting key ae90e7ecf14fb3ba54e5da36a1ed8995
I20260620 00:16:32.208527 19596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:32.208633 19596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280cee93be0a4294bb002af078caa4df, State: Initialized, Role: FOLLOWER
I20260620 00:16:32.208828 19596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } }
I20260620 00:16:32.208935 19596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:32.208976 19596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:32.209120 19596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:32.210685 19596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } }
I20260620 00:16:32.210839 19596 leader_election.cc:302] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280cee93be0a4294bb002af078caa4df; no voters:
I20260620 00:16:32.211079 19596 leader_election.cc:288] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:32.211110 19601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:32.211229 19601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [term 1 LEADER]: Becoming Leader. State: Replica: 280cee93be0a4294bb002af078caa4df, State: Running, Role: LEADER
I20260620 00:16:32.211319 19601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } }
I20260620 00:16:32.211387 19596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:32.211768 19602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280cee93be0a4294bb002af078caa4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } } }
I20260620 00:16:32.211860 19602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:32.212110 19603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280cee93be0a4294bb002af078caa4df. Latest consensus state: current_term: 1 leader_uuid: "280cee93be0a4294bb002af078caa4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280cee93be0a4294bb002af078caa4df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43087 } } }
I20260620 00:16:32.212208 19603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:32.212402 19607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:32.213065 19607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:32.215173 19607 catalog_manager.cc:1347] Generated new cluster ID: ac60bb7af2d2458ab178b6a4ee542798
I20260620 00:16:32.215234 19607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:32.222259 19607 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:32.222822 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.209050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.98.62:43087)
0620 00:16:32.209263 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.209272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.209482 (+ 210us) server_negotiation.cc:407] Connection header received
0620 00:16:32.209663 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.209669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.209734 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.209860 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:32.210905 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.212360 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.213614 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.214173 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.214319 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.217543 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:32.217549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:32.217549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:32.217571 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:32.220226 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.220855 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.220860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.220860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.221014 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.221287 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.221288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.221289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.221659 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:32.222279 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.222371 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.222576 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260620 00:16:32.223125 19607 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:32.236085 19607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: Generated new TSK 0
I20260620 00:16:32.236352 19607 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:16:32.272884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:32.357391 19622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:32.357630 19622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:32.357657 19622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:32.358934 19622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:32.358987 19622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:32.359018 19622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:32.360677 19622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:32.360953 19622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:32.361200 19622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:32.366568 19630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:32.366611 19628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.366701 19622 server_base.cc:1034] running on GCE node
W20260620 00:16:32.366603 19627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.367100 19622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:32.367326 19622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:32.368523 19622 hybrid_clock.cc:648] HybridClock initialized: now 1781914592368506 us; error 34 us; skew 500 ppm
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:32.371270 19622 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:16:32.371776 19622 webserver.cc:460] Webserver started at http://127.0.98.1:34629/ using document root <none> and password file <none>
I20260620 00:16:32.372022 19622 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:32.372092 19622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:32.372253 19622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:32.373423 19622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "a624793a012c4145957f99224fc015a1"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "07c4d95ca9d8b3a3045586fc43278d8d"
server_key_iv: "c1632fc0de0a78b87441c1c508017ffa"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.374053 19622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a624793a012c4145957f99224fc015a1"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "07c4d95ca9d8b3a3045586fc43278d8d"
server_key_iv: "c1632fc0de0a78b87441c1c508017ffa"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.375823 19622 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:16:32.376746 19637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.376986 19622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:32.377102 19622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "a624793a012c4145957f99224fc015a1"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "07c4d95ca9d8b3a3045586fc43278d8d"
server_key_iv: "c1632fc0de0a78b87441c1c508017ffa"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.377202 19622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:32.387171 19622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:32.389668 19622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:32.389846 19622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:32.390098 19622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:32.390442 19622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:32.390506 19622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.390542 19622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:32.390578 19622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.396855 19622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46743
I20260620 00:16:32.396896 19750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46743 every 8 connection(s)
I20260620 00:16:32.397302 19622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:32.398231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19622
I20260620 00:16:32.398366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:32.398658 392 external_mini_cluster.cc:1428] Setting key 2deef37683f299892e7facd6690da7a7
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:16:32.409628 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.398164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47843 (local address 127.0.98.62:43087)
0620 00:16:32.398216 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.398218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.398401 (+ 183us) server_negotiation.cc:407] Connection header received
0620 00:16:32.399796 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.399797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.399818 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.399876 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:32.400844 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.401761 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.402937 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.403282 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.403374 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.406182 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:32.406184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:32.406185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:32.406195 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:32.407983 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.408606 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.408608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.408608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.408683 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.408939 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.408939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.408940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.409090 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:32.409149 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.409477 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.409555 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:16:32.409868 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.398034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43087 (local address 127.0.98.1:47843)
0620 00:16:32.398250 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.398305 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.399654 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.399958 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.399963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.400403 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:32.400737 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.400743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.401897 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.401898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.402821 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.402824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.403404 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.403406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.403599 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.404319 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:32.404329 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:32.406056 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:32.408115 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.408128 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.408131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.408403 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.408758 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.408759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.408759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.408885 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.409190 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:32.409197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:32.409344 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.409450 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.409596 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260620 00:16:32.410506 19751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43087
I20260620 00:16:32.410648 19751 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:32.410887 19751 heartbeater.cc:511] Master 127.0.98.62:43087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:16:32.411515 19560 ts_manager.cc:194] Registered new tserver with Master: a624793a012c4145957f99224fc015a1 (127.0.98.1:46743)
I20260620 00:16:32.412586 19560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:16:32.428263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:32.518239 19756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:32.518507 19756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:32.518550 19756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:32.519969 19756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:32.520026 19756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:32.520057 19756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:32.521750 19756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:32.521981 19756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:32.522179 19756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:32.527878 19761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:32.527935 19762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.527940 19756 server_base.cc:1034] running on GCE node
W20260620 00:16:32.527971 19764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.528393 19756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:32.528774 19756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:32.530020 19756 hybrid_clock.cc:648] HybridClock initialized: now 1781914592530008 us; error 33 us; skew 500 ppm
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:32.533632 19756 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:16:32.534250 19756 webserver.cc:460] Webserver started at http://127.0.98.2:39621/ using document root <none> and password file <none>
I20260620 00:16:32.534526 19756 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:32.534600 19756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:32.534763 19756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:32.535924 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "bf024af9a4244a77b5ee804658860063"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "f70c45612f1df4a1bdf130fe2a8637b0"
server_key_iv: "ee596956300f3b655db470428f98ba24"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.536350 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bf024af9a4244a77b5ee804658860063"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "f70c45612f1df4a1bdf130fe2a8637b0"
server_key_iv: "ee596956300f3b655db470428f98ba24"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.538152 19756 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:16:32.539291 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.539594 19756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:32.539747 19756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "bf024af9a4244a77b5ee804658860063"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "f70c45612f1df4a1bdf130fe2a8637b0"
server_key_iv: "ee596956300f3b655db470428f98ba24"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.539846 19756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:32.559018 19756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:32.561908 19756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:32.562036 19756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:32.562286 19756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:32.562651 19756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:32.562704 19756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.562738 19756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:32.562757 19756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.568519 19756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36803
I20260620 00:16:32.568632 19884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36803 every 8 connection(s)
I20260620 00:16:32.569007 19756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:32.574162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19756
I20260620 00:16:32.574272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:32.574473 392 external_mini_cluster.cc:1428] Setting key dd266f4b0537de8b97db1ad400ac1d9a
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:32.580166 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.569889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44385 (local address 127.0.98.62:43087)
0620 00:16:32.569980 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.569982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.570116 (+ 134us) server_negotiation.cc:407] Connection header received
0620 00:16:32.571104 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.571104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.571114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.571179 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:32.571801 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.572311 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.573452 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.573638 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.573695 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.576637 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:32.576639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:32.576639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:32.576648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:32.578410 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.578948 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.578950 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.578950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.579019 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.579374 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.579375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.579375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.579531 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:32.579610 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.579884 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.579989 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 00:16:32.580171 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.569794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43087 (local address 127.0.98.2:44385)
0620 00:16:32.569980 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.570025 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.570949 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.571253 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.571256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.571527 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:32.571727 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.571731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.572419 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.572420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.573373 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.573375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.573738 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.573739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.573848 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.574543 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:32.574551 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:32.576516 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:32.578571 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.578580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.578583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.578854 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.579120 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.579122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.579122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.579291 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.579603 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:32.579609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:32.579743 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.579834 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.579987 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260620 00:16:32.580854 19885 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43087
I20260620 00:16:32.580971 19885 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:32.581174 19885 heartbeater.cc:511] Master 127.0.98.62:43087 requested a full tablet report, sending...
I20260620 00:16:32.581688 19560 ts_manager.cc:194] Registered new tserver with Master: bf024af9a4244a77b5ee804658860063 (127.0.98.2:36803)
I20260620 00:16:32.582248 19560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44385
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:16:32.605646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:32.695031 19890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:32.695290 19890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:32.695320 19890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:32.696784 19890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:32.696838 19890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:32.696869 19890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:32.698490 19890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:32.698755 19890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:32.699048 19890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:32.705070 19898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.705053 19890 server_base.cc:1034] running on GCE node
W20260620 00:16:32.705053 19895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:32.705092 19896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:32.705574 19890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:32.705855 19890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:32.707031 19890 hybrid_clock.cc:648] HybridClock initialized: now 1781914592707006 us; error 39 us; skew 500 ppm
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:32.709935 19890 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:16:32.710474 19890 webserver.cc:460] Webserver started at http://127.0.98.3:32789/ using document root <none> and password file <none>
I20260620 00:16:32.710726 19890 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:32.710791 19890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:32.710958 19890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:32.712193 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "b6967cd3df4542e193a4cc868ce9a330"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "7d4315f0c5e9cb6371b45e6bb1047a7f"
server_key_iv: "b6e8280ad9a5887ce38fcc08592063f3"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.712615 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b6967cd3df4542e193a4cc868ce9a330"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "7d4315f0c5e9cb6371b45e6bb1047a7f"
server_key_iv: "b6e8280ad9a5887ce38fcc08592063f3"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.714161 19890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:32.714975 19905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.715261 19890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:32.715348 19890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "b6967cd3df4542e193a4cc868ce9a330"
format_stamp: "Formatted at 2026-06-20 00:16:32 on dist-test-slave-28bv"
server_key: "7d4315f0c5e9cb6371b45e6bb1047a7f"
server_key_iv: "b6e8280ad9a5887ce38fcc08592063f3"
server_key_version: "encryptionkey@0"
I20260620 00:16:32.715409 19890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:32.724893 19890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:32.727212 19890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:32.727345 19890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:32.727596 19890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:32.727895 19890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:32.727924 19890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.727948 19890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:32.727962 19890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:32.734517 19890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46005
I20260620 00:16:32.734558 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46005 every 8 connection(s)
I20260620 00:16:32.734922 19890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:32.741567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 19890
I20260620 00:16:32.741668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:32.741874 392 external_mini_cluster.cc:1428] Setting key 57693fdaefc3e1495b9e74419b2e5055
Jun 20 00:16:32 dist-test-slave-28bv krb5kdc[19515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:32.746745 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.735782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39197 (local address 127.0.98.62:43087)
0620 00:16:32.735913 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.735915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.736124 (+ 209us) server_negotiation.cc:407] Connection header received
0620 00:16:32.737363 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.737364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.737376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.737453 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:32.738152 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.738660 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.739728 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.739929 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.739999 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.743055 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:32.743057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:32.743057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:32.743067 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:32.745205 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.745698 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.745699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.745699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.745761 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.746092 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.746093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.746093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.746217 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:32.746276 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.746550 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.746654 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 00:16:32.746995 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.735752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43087 (local address 127.0.98.3:39197)
0620 00:16:32.735959 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.736004 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.737225 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.737531 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.737534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.737819 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:32.738022 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.738026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.738751 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.738752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.739626 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.739628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.740055 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.740056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.740182 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.740912 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:32.740921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:32.742798 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:32.745307 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.745317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.745319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.745600 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.745866 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:32.745866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:32.745867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:32.745995 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:32.746283 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:32.746288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:32.746425 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.746516 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.746693 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":139,"threads_started":1}
I20260620 00:16:32.747710 20019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43087
I20260620 00:16:32.747812 20019 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:32.748039 20019 heartbeater.cc:511] Master 127.0.98.62:43087 requested a full tablet report, sending...
I20260620 00:16:32.748616 19560 ts_manager.cc:194] Registered new tserver with Master: b6967cd3df4542e193a4cc868ce9a330 (127.0.98.3:46005)
I20260620 00:16:32.749069 19560 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39197
I20260620 00:16:32.754136 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:32.762921 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.755605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.0.98.62:43087)
0620 00:16:32.755718 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.755720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.755824 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:16:32.755931 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.755932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.755945 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.756021 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:32.756940 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.757445 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.758193 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.758402 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.758479 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.760047 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:32.760048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:32.760049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:32.760084 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:32.761620 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.762039 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.762040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.762040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.762107 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:32.762307 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:32.762308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:32.762309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:32.762475 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:32.762603 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.762723 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.762812 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 00:16:32.764943 19560 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:32.766618 19560 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:32.773214 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.0.98.1:46743)
0620 00:16:32.769405 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.769410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.769420 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:32.769462 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.769464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.769482 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.769567 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:32.769833 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.770728 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.772000 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.772794 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.772865 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.772927 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.773077 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.773118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":101,"threads_started":1}
I20260620 00:16:32.773207 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46743 (local address 127.0.0.1:52116)
0620 00:16:32.769230 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.769276 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.769343 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.769591 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.769594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.769597 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:32.769735 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.769738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.770837 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.770839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.771899 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.771902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.772876 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.772877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.772957 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.772972 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.773052 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.773117 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":55,"threads_started":1}
I20260620 00:16:32.773717 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46005 (local address 127.0.0.1:39410)
0620 00:16:32.769467 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.769476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.769516 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.769987 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.769988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.769990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:32.770036 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.770038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.771467 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.771468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.772518 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.772520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.773482 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.773483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.773546 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.773551 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.773610 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.773648 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":63,"threads_started":1}
I20260620 00:16:32.773742 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.98.3:46005)
0620 00:16:32.769830 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.769834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.769844 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:32.769878 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.769880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.769896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.769976 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:32.770178 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.771355 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.772653 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.773386 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.773482 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.773536 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.773622 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.773666 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":156,"threads_started":1}
I20260620 00:16:32.773787 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.98.2:36803)
0620 00:16:32.769694 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.769698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.769710 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:16:32.769812 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.769814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.770133 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.770209 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:32.770434 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.771467 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.772587 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.773221 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.773286 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.773342 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.773530 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.773586 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":87,"threads_started":1}
I20260620 00:16:32.774394 19953 tablet_service.cc:1467] Processing CreateTablet for tablet 276410f835ec464cb68fef35e4bde4e7 (DEFAULT_TABLE table=test-table [id=f94a0061c98e4c54b3353005d760c418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:32.774394 19685 tablet_service.cc:1467] Processing CreateTablet for tablet 276410f835ec464cb68fef35e4bde4e7 (DEFAULT_TABLE table=test-table [id=f94a0061c98e4c54b3353005d760c418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:32.774770 19685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276410f835ec464cb68fef35e4bde4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:32.774781 19953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276410f835ec464cb68fef35e4bde4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:32.777025 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.769076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36803 (local address 127.0.0.1:45274)
0620 00:16:32.769393 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.769405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.769706 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.770294 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.770295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.770297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:32.770337 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.770338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.771570 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.771570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.772479 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.772482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.773382 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.773383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.773449 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.773453 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.776832 (+ 3379us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.776911 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":86,"threads_started":1}
I20260620 00:16:32.778050 20036 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Bootstrap starting.
I20260620 00:16:32.777967 19819 tablet_service.cc:1467] Processing CreateTablet for tablet 276410f835ec464cb68fef35e4bde4e7 (DEFAULT_TABLE table=test-table [id=f94a0061c98e4c54b3353005d760c418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:32.778335 19819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 276410f835ec464cb68fef35e4bde4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:32.778393 20037 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Bootstrap starting.
I20260620 00:16:32.778807 20036 tablet_bootstrap.cc:654] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:32.779003 20037 tablet_bootstrap.cc:654] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:32.779433 20036 log.cc:824] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:32.779673 20037 log.cc:824] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:32.780493 20036 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: No bootstrap required, opened a new log
I20260620 00:16:32.780515 20037 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: No bootstrap required, opened a new log
I20260620 00:16:32.780606 20036 ts_tablet_manager.cc:1397] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:16:32.780629 20037 ts_tablet_manager.cc:1397] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:16:32.781239 20040 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Bootstrap starting.
I20260620 00:16:32.781913 20040 tablet_bootstrap.cc:654] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:32.782502 20040 log.cc:824] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:32.782735 20037 raft_consensus.cc:348] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.782943 20037 raft_consensus.cc:374] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:32.782982 20037 raft_consensus.cc:729] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6967cd3df4542e193a4cc868ce9a330, State: Initialized, Role: FOLLOWER
I20260620 00:16:32.782876 20036 raft_consensus.cc:348] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.783026 20036 raft_consensus.cc:374] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:32.783068 20036 raft_consensus.cc:729] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a624793a012c4145957f99224fc015a1, State: Initialized, Role: FOLLOWER
I20260620 00:16:32.783114 20037 consensus_queue.cc:260] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.783201 20036 consensus_queue.cc:260] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.783439 20037 ts_tablet_manager.cc:1428] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:16:32.783461 20036 ts_tablet_manager.cc:1428] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:16:32.783571 20019 heartbeater.cc:503] Master 127.0.98.62:43087 was elected leader, sending a full tablet report...
I20260620 00:16:32.783581 19751 heartbeater.cc:503] Master 127.0.98.62:43087 was elected leader, sending a full tablet report...
I20260620 00:16:32.783850 20040 tablet_bootstrap.cc:492] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: No bootstrap required, opened a new log
I20260620 00:16:32.784137 20040 ts_tablet_manager.cc:1397] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:16:32.786015 20040 raft_consensus.cc:348] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.786201 20040 raft_consensus.cc:374] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:32.786254 20040 raft_consensus.cc:729] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf024af9a4244a77b5ee804658860063, State: Initialized, Role: FOLLOWER
I20260620 00:16:32.786377 20040 consensus_queue.cc:260] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.786731 20040 ts_tablet_manager.cc:1428] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:16:32.786808 19885 heartbeater.cc:503] Master 127.0.98.62:43087 was elected leader, sending a full tablet report...
W20260620 00:16:32.819573 19886 tablet.cc:2369] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:32.871081 20044 raft_consensus.cc:482] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:32.871217 20044 raft_consensus.cc:504] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.871542 20044 leader_election.cc:288] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a624793a012c4145957f99224fc015a1 (127.0.98.1:46743), b6967cd3df4542e193a4cc868ce9a330 (127.0.98.3:46005)
I20260620 00:16:32.874641 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.871783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46005 (local address 127.0.98.2:46451)
0620 00:16:32.871854 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.871869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.871936 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.872125 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.872126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.872128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:32.872203 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.872204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.872914 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.872915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.873702 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.873704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.874327 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.874328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.874407 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.874411 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.874487 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.874529 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 00:16:32.874619 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.871886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46451 (local address 127.0.98.3:46005)
0620 00:16:32.871984 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.871986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.871995 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:32.872025 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.872026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.872037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.872105 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:32.872277 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.872822 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.873813 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.874229 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.874314 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.874326 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.874492 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.874533 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:32.874644 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.871919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41463 (local address 127.0.98.1:46743)
0620 00:16:32.871977 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.871979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.872134 (+ 155us) server_negotiation.cc:407] Connection header received
0620 00:16:32.872242 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.872243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.872257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.872332 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:32.872495 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.872976 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.873911 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.874272 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.874356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.874365 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.874527 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.874566 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:16:32.874660 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.871783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46743 (local address 127.0.98.2:41463)
0620 00:16:32.872052 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.872061 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:32.872165 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:32.872335 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:32.872336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:32.872337 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:32.872424 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.872426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.873063 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.873065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.873816 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:32.873818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.874359 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.874360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:32.874449 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.874452 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:32.874521 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:16:32.874563 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":74,"thread_start_us":118,"threads_started":1}
I20260620 00:16:32.875012 19973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276410f835ec464cb68fef35e4bde4e7" candidate_uuid: "bf024af9a4244a77b5ee804658860063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6967cd3df4542e193a4cc868ce9a330" is_pre_election: true
I20260620 00:16:32.875008 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276410f835ec464cb68fef35e4bde4e7" candidate_uuid: "bf024af9a4244a77b5ee804658860063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a624793a012c4145957f99224fc015a1" is_pre_election: true
I20260620 00:16:32.875192 19973 raft_consensus.cc:2454] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf024af9a4244a77b5ee804658860063 in term 0.
I20260620 00:16:32.875221 19705 raft_consensus.cc:2454] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf024af9a4244a77b5ee804658860063 in term 0.
I20260620 00:16:32.875427 19772 leader_election.cc:302] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6967cd3df4542e193a4cc868ce9a330, bf024af9a4244a77b5ee804658860063; no voters:
I20260620 00:16:32.875662 20044 raft_consensus.cc:2781] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:32.875764 20044 raft_consensus.cc:482] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:32.875788 20044 raft_consensus.cc:3037] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:32.876761 20044 raft_consensus.cc:504] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.876919 20044 leader_election.cc:288] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [CANDIDATE]: Term 1 election: Requested vote from peers a624793a012c4145957f99224fc015a1 (127.0.98.1:46743), b6967cd3df4542e193a4cc868ce9a330 (127.0.98.3:46005)
I20260620 00:16:32.877179 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276410f835ec464cb68fef35e4bde4e7" candidate_uuid: "bf024af9a4244a77b5ee804658860063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a624793a012c4145957f99224fc015a1"
I20260620 00:16:32.877229 19973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276410f835ec464cb68fef35e4bde4e7" candidate_uuid: "bf024af9a4244a77b5ee804658860063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6967cd3df4542e193a4cc868ce9a330"
I20260620 00:16:32.877287 19705 raft_consensus.cc:3037] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:32.877303 19973 raft_consensus.cc:3037] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:32.878193 19973 raft_consensus.cc:2454] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf024af9a4244a77b5ee804658860063 in term 1.
I20260620 00:16:32.878193 19705 raft_consensus.cc:2454] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf024af9a4244a77b5ee804658860063 in term 1.
I20260620 00:16:32.878448 19775 leader_election.cc:302] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a624793a012c4145957f99224fc015a1, bf024af9a4244a77b5ee804658860063; no voters:
I20260620 00:16:32.878558 20044 raft_consensus.cc:2781] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:32.878754 20044 raft_consensus.cc:686] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 1 LEADER]: Becoming Leader. State: Replica: bf024af9a4244a77b5ee804658860063, State: Running, Role: LEADER
I20260620 00:16:32.878867 20044 consensus_queue.cc:237] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } }
I20260620 00:16:32.879771 19559 catalog_manager.cc:5495] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf024af9a4244a77b5ee804658860063 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bf024af9a4244a77b5ee804658860063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf024af9a4244a77b5ee804658860063" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 } health_report { overall_health: UNKNOWN } } }
I20260620 00:16:32.894189 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.891837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.0.98.2:36803)
0620 00:16:32.891895 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:32.891897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:32.891907 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:32.891943 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:32.891943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:32.891963 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:32.892027 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:32.892125 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.892691 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.893419 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:32.893634 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:32.893688 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:32.893836 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:32.893941 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:32.894002 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:32.894032 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 00:16:32.894052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:16:32.896021 19705 raft_consensus.cc:1261] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 1 FOLLOWER]: Refusing update from remote peer bf024af9a4244a77b5ee804658860063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:32.896042 19973 raft_consensus.cc:1261] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 1 FOLLOWER]: Refusing update from remote peer bf024af9a4244a77b5ee804658860063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:32.896361 20044 consensus_queue.cc:1035] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a624793a012c4145957f99224fc015a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:32.896503 20046 consensus_queue.cc:1035] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6967cd3df4542e193a4cc868ce9a330" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:16:32.898036 19752 tablet.cc:2369] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:32.899408 20049 mvcc.cc:205] Tried to move back new op lower bound from 7298722172499410944 to 7298722172432642048. Current Snapshot: MvccSnapshot[applied={T|T < 7298722172499410944}]
I20260620 00:16:32.899813 20051 mvcc.cc:205] Tried to move back new op lower bound from 7298722172499410944 to 7298722172432642048. Current Snapshot: MvccSnapshot[applied={T|T < 7298722172499410944}]
I20260620 00:16:32.899829 20050 mvcc.cc:205] Tried to move back new op lower bound from 7298722172499410944 to 7298722172432642048. Current Snapshot: MvccSnapshot[applied={T|T < 7298722172499410944}]
I20260620 00:16:32.902081 19559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:32.902181 19559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:32.903817 19559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 280cee93be0a4294bb002af078caa4df: Sending DeleteTablet for 3 replicas of tablet 276410f835ec464cb68fef35e4bde4e7
I20260620 00:16:32.904240 19819 tablet_service.cc:1514] Processing DeleteTablet for tablet 276410f835ec464cb68fef35e4bde4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260620 00:16:32.904260 19685 tablet_service.cc:1514] Processing DeleteTablet for tablet 276410f835ec464cb68fef35e4bde4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52116
I20260620 00:16:32.904251 19953 tablet_service.cc:1514] Processing DeleteTablet for tablet 276410f835ec464cb68fef35e4bde4e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39410
I20260620 00:16:32.904575 20060 tablet_replica.cc:330] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1: stopping tablet replica
I20260620 00:16:32.904690 20060 raft_consensus.cc:2229] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:32.904779 20061 tablet_replica.cc:330] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: stopping tablet replica
I20260620 00:16:32.904808 20060 raft_consensus.cc:2258] T 276410f835ec464cb68fef35e4bde4e7 P a624793a012c4145957f99224fc015a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:32.904830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19622
I20260620 00:16:32.908711 20061 raft_consensus.cc:2229] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:32.908859 20061 raft_consensus.cc:2258] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:32.909042 20059 tablet_replica.cc:330] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: stopping tablet replica
I20260620 00:16:32.909148 20059 raft_consensus.cc:2229] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:16:32.909251 20059 raft_consensus.cc:2258] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:32.909431 20061 ts_tablet_manager.cc:1905] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:32.909752 20059 ts_tablet_manager.cc:1905] T 276410f835ec464cb68fef35e4bde4e7 P bf024af9a4244a77b5ee804658860063: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:16:32.911571 19547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46743 (error 108)
I20260620 00:16:32.911914 20061 ts_tablet_manager.cc:1918] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:32.911918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19756
I20260620 00:16:32.912047 20061 log.cc:1196] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/276410f835ec464cb68fef35e4bde4e7
I20260620 00:16:32.912312 20061 ts_tablet_manager.cc:1939] T 276410f835ec464cb68fef35e4bde4e7 P b6967cd3df4542e193a4cc868ce9a330: Deleting consensus metadata
I20260620 00:16:32.912901 19544 catalog_manager.cc:4841] TS b6967cd3df4542e193a4cc868ce9a330 (127.0.98.3:46005): tablet 276410f835ec464cb68fef35e4bde4e7 (table test-table [id=f94a0061c98e4c54b3353005d760c418]) successfully deleted
I20260620 00:16:32.913502 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.913366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46743 (local address 127.0.0.1:52118)
0620 00:16:32.913427 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.913465 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:16:32.913676 19547 catalog_manager.cc:4584] TS a624793a012c4145957f99224fc015a1 (127.0.98.1:46743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 276410f835ec464cb68fef35e4bde4e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46743: connect: Connection refused (error 111)
I20260620 00:16:32.917585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19890
I20260620 00:16:32.917610 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:32.917456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36803 (local address 127.0.0.1:45294)
0620 00:16:32.917539 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:32.917568 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 00:16:32.922446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 19528
2026-06-20T00:16:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T00:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:32Z Disabled control of system clock
I20260620 00:16:32.951093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34535
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:33.034642 20066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:33.034885 20066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:33.034907 20066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:33.036051 20066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:33.036088 20066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:33.036101 20066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:33.036114 20066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:33.036126 20066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:33.037451 20066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:33.037729 20066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:33.037987 20066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:33.043334 20071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.043486 20066 server_base.cc:1034] running on GCE node
W20260620 00:16:33.043335 20072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:33.043454 20074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.043875 20066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:33.044169 20066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:33.045331 20066 hybrid_clock.cc:648] HybridClock initialized: now 1781914593045308 us; error 39 us; skew 500 ppm
I20260620 00:16:33.046815 20066 webserver.cc:460] Webserver started at http://127.0.98.62:36555/ using document root <none> and password file <none>
I20260620 00:16:33.047086 20066 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:33.047166 20066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:33.047318 20066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:33.048435 20066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "98d5a720a5c244ce862401d32c0affc9"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "efd7fd4e327f0f2a4faf6fa50aaf6a2f"
server_key_iv: "043e17b4232a2a32da4589f06b397245"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.048893 20066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "98d5a720a5c244ce862401d32c0affc9"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "efd7fd4e327f0f2a4faf6fa50aaf6a2f"
server_key_iv: "043e17b4232a2a32da4589f06b397245"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.050299 20066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:33.051249 20080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.051504 20066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:33.051633 20066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "98d5a720a5c244ce862401d32c0affc9"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "efd7fd4e327f0f2a4faf6fa50aaf6a2f"
server_key_iv: "043e17b4232a2a32da4589f06b397245"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.051712 20066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:33.066452 20066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:33.066809 20066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:33.066983 20066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:33.071278 20066 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34535
I20260620 00:16:33.071310 20132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34535 every 8 connection(s)
I20260620 00:16:33.071681 20066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:33.072726 20133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:33.075076 20133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: Bootstrap starting.
I20260620 00:16:33.075748 20133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:33.076014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20066
I20260620 00:16:33.076068 20133 log.cc:824] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:33.076130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:33.076387 392 external_mini_cluster.cc:1428] Setting key c5fdd764185525006585458f20854005
I20260620 00:16:33.077057 20133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: No bootstrap required, opened a new log
I20260620 00:16:33.078825 20133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260620 00:16:33.078998 20133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:33.079038 20133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d5a720a5c244ce862401d32c0affc9, State: Initialized, Role: FOLLOWER
I20260620 00:16:33.079200 20133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260620 00:16:33.079313 20133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:33.079356 20133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:33.079411 20133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:33.080011 20133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260620 00:16:33.080140 20133 leader_election.cc:302] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98d5a720a5c244ce862401d32c0affc9; no voters:
I20260620 00:16:33.080363 20133 leader_election.cc:288] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:33.080402 20138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:33.080507 20138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [term 1 LEADER]: Becoming Leader. State: Replica: 98d5a720a5c244ce862401d32c0affc9, State: Running, Role: LEADER
I20260620 00:16:33.080611 20138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } }
I20260620 00:16:33.080689 20133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:33.080950 20138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98d5a720a5c244ce862401d32c0affc9. Latest consensus state: current_term: 1 leader_uuid: "98d5a720a5c244ce862401d32c0affc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } } }
I20260620 00:16:33.081032 20138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:33.080976 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98d5a720a5c244ce862401d32c0affc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98d5a720a5c244ce862401d32c0affc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34535 } } }
I20260620 00:16:33.081215 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:33.081243 20142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:33.081862 20142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:33.084764 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.077013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.0.98.62:34535)
0620 00:16:33.077217 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.077232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.077257 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:16:33.078821 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.078830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.078900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.079086 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:33.079326 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.081127 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.082364 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.083139 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.083299 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.084067 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:33.084072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:33.084072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:33.084244 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:33.084281 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:33.084292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:33.084377 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.084453 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.084568 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260620 00:16:33.084911 20142 catalog_manager.cc:1347] Generated new cluster ID: c46c65cda6ff48d19d10f3cdf710b461
I20260620 00:16:33.084956 20142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:33.092686 20142 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:33.093490 20142 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:33.099797 20142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: Generated new TSK 0
I20260620 00:16:33.100081 20142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:33.108611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:16:33.197060 20157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:33.197310 20157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:33.197335 20157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:33.198633 20157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:33.198685 20157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:33.198714 20157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:16:33.200181 20157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:33.200455 20157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:33.200815 20157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:33.206171 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:33.206161 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.206374 20157 server_base.cc:1034] running on GCE node
W20260620 00:16:33.206171 20162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.206753 20157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:33.207063 20157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:33.208240 20157 hybrid_clock.cc:648] HybridClock initialized: now 1781914593208214 us; error 43 us; skew 500 ppm
I20260620 00:16:33.210270 20157 webserver.cc:460] Webserver started at http://127.0.98.1:34403/ using document root <none> and password file <none>
I20260620 00:16:33.210559 20157 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:33.210611 20157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:33.210716 20157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:33.211812 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data/instance:
uuid: "1e44d0a667fc42cd95cdc18c27738b05"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "3ec984f945a3ae65a77b807a5d53de5e"
server_key_iv: "314ced20029aca00ba0ba42d892b38a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.212199 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1e44d0a667fc42cd95cdc18c27738b05"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "3ec984f945a3ae65a77b807a5d53de5e"
server_key_iv: "314ced20029aca00ba0ba42d892b38a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.213711 20157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:33.214578 20171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.214781 20157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:33.214877 20157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal
uuid: "1e44d0a667fc42cd95cdc18c27738b05"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "3ec984f945a3ae65a77b807a5d53de5e"
server_key_iv: "314ced20029aca00ba0ba42d892b38a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.214955 20157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:33.228760 20157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:33.229099 20157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:33.229247 20157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:33.229539 20157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:33.229892 20157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:33.229928 20157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.229952 20157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:33.229993 20157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.236392 20157 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42359
I20260620 00:16:33.236434 20284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42359 every 8 connection(s)
I20260620 00:16:33.236886 20157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:16:33.243292 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.237808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49383 (local address 127.0.98.62:34535)
0620 00:16:33.237895 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.237896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.238132 (+ 236us) server_negotiation.cc:407] Connection header received
0620 00:16:33.238807 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.238808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.238823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.238896 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:33.239761 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.240317 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.241731 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.241945 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.242012 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.243045 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:33.243046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:33.243046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:33.243052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:33.243067 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:33.243071 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:33.243155 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.243196 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.243226 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:16:33.243597 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.237773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.1:49383)
0620 00:16:33.237983 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.238031 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.238630 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.239096 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.239100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.239110 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:33.239642 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.239647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.240431 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.240432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.241571 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.241574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.242054 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.242055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.242185 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.242839 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:33.242840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:33.242909 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:33.242937 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:33.243139 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:33.243143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:33.243145 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.243198 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.243287 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260620 00:16:33.244232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20157
I20260620 00:16:33.244234 20285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260620 00:16:33.244341 20285 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:33.244354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:16:33.244575 392 external_mini_cluster.cc:1428] Setting key 14e3aed36f89844f8d51aa507779f474
I20260620 00:16:33.244642 20285 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
I20260620 00:16:33.245211 20096 ts_manager.cc:194] Registered new tserver with Master: 1e44d0a667fc42cd95cdc18c27738b05 (127.0.98.1:42359)
I20260620 00:16:33.245546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:16:33.245936 20096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49383
W20260620 00:16:33.328712 20288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:33.328953 20288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:33.328979 20288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:33.330257 20288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:33.330315 20288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:33.330343 20288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:16:33.331815 20288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:33.332103 20288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:33.332427 20288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:33.337849 20293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:33.337901 20296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.338047 20288 server_base.cc:1034] running on GCE node
W20260620 00:16:33.337934 20294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.338399 20288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:33.338683 20288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:33.339864 20288 hybrid_clock.cc:648] HybridClock initialized: now 1781914593339851 us; error 36 us; skew 500 ppm
I20260620 00:16:33.341341 20288 webserver.cc:460] Webserver started at http://127.0.98.2:35721/ using document root <none> and password file <none>
I20260620 00:16:33.341611 20288 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:33.341684 20288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:33.341841 20288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:33.342967 20288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data/instance:
uuid: "fbe6daecd6bd43eca446e9bdac52f326"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "176c79d71f0edd9cdf7df34ddc039319"
server_key_iv: "bad516a61a9c0df825272f138969cc28"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.343328 20288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe6daecd6bd43eca446e9bdac52f326"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "176c79d71f0edd9cdf7df34ddc039319"
server_key_iv: "bad516a61a9c0df825272f138969cc28"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.344702 20288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:33.345513 20302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.345685 20288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:33.345782 20288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal
uuid: "fbe6daecd6bd43eca446e9bdac52f326"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "176c79d71f0edd9cdf7df34ddc039319"
server_key_iv: "bad516a61a9c0df825272f138969cc28"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.345865 20288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:33.363099 20288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:33.363435 20288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:33.363588 20288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:33.363847 20288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:33.364183 20288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:33.364238 20288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.364282 20288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:33.364307 20288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.370370 20288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34167
I20260620 00:16:33.370425 20415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34167 every 8 connection(s)
I20260620 00:16:33.370784 20288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:16:33.371160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20288
I20260620 00:16:33.371289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:16:33.371502 392 external_mini_cluster.cc:1428] Setting key 3d4653fd3524f7b6f557d967f629b933
I20260620 00:16:33.372846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--builtin_ntp_servers=127.0.98.20:39911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:16:33.376816 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.371589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49259 (local address 127.0.98.62:34535)
0620 00:16:33.371669 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.371672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.371977 (+ 305us) server_negotiation.cc:407] Connection header received
0620 00:16:33.372655 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.372656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.372668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.372721 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:33.373740 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.374274 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.375272 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.375487 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.375560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.376492 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:33.376493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:33.376494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:33.376499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:33.376513 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:33.376517 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:33.376612 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.376703 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.376733 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:16:33.377049 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.371605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.2:49259)
0620 00:16:33.371807 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.371850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.372484 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.372802 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.372806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.372823 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:33.373596 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.373603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.374374 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.374375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.375179 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.375181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.375582 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.375583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.375722 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.376320 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:33.376320 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:33.376349 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:33.376376 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:33.376609 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:33.376614 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:33.376617 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.376707 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.376814 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260620 00:16:33.377489 20416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260620 00:16:33.377584 20416 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:33.377795 20416 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
I20260620 00:16:33.378182 20096 ts_manager.cc:194] Registered new tserver with Master: fbe6daecd6bd43eca446e9bdac52f326 (127.0.98.2:34167)
I20260620 00:16:33.378554 20096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49259
W20260620 00:16:33.457747 20419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:33.458002 20419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:33.458043 20419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:33.459293 20419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:33.459367 20419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:33.459414 20419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:16:33.460974 20419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:33.461257 20419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:33.461573 20419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:33.466631 20424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:33.466646 20425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:33.466646 20427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:33.466876 20419 server_base.cc:1034] running on GCE node
I20260620 00:16:33.467146 20419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:33.467410 20419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:33.468667 20419 hybrid_clock.cc:648] HybridClock initialized: now 1781914593468635 us; error 59 us; skew 500 ppm
I20260620 00:16:33.470139 20419 webserver.cc:460] Webserver started at http://127.0.98.3:41127/ using document root <none> and password file <none>
I20260620 00:16:33.470420 20419 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:33.470496 20419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:33.470659 20419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:33.471751 20419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data/instance:
uuid: "a975b7e7f00742588f669d9bffe6e514"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "1912a20355b9dfd7283e9a5fade5bac2"
server_key_iv: "4e29e2369956ba8502bc7621e1ec2683"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.472133 20419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a975b7e7f00742588f669d9bffe6e514"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "1912a20355b9dfd7283e9a5fade5bac2"
server_key_iv: "4e29e2369956ba8502bc7621e1ec2683"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.473548 20419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:33.474370 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.474589 20419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:33.474707 20419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal
uuid: "a975b7e7f00742588f669d9bffe6e514"
format_stamp: "Formatted at 2026-06-20 00:16:33 on dist-test-slave-28bv"
server_key: "1912a20355b9dfd7283e9a5fade5bac2"
server_key_iv: "4e29e2369956ba8502bc7621e1ec2683"
server_key_version: "encryptionkey@0"
I20260620 00:16:33.474790 20419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:33.497113 20419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:33.497459 20419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:33.497618 20419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:33.497870 20419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:16:33.498198 20419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:16:33.498251 20419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.498294 20419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:16:33.498319 20419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:33.504319 20419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42187
I20260620 00:16:33.504402 20546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42187 every 8 connection(s)
I20260620 00:16:33.504769 20419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:16:33.508404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20419
I20260620 00:16:33.508508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:16:33.508780 392 external_mini_cluster.cc:1428] Setting key 333888297f93f5fd0214b07587cf90e8
I20260620 00:16:33.510352 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.505588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56043 (local address 127.0.98.62:34535)
0620 00:16:33.505659 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.505661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.505889 (+ 228us) server_negotiation.cc:407] Connection header received
0620 00:16:33.506436 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.506438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.506450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.506498 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:33.507195 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.507704 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.508816 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.509074 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.509166 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.510066 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:33.510068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:33.510069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:33.510074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:33.510087 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:33.510092 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:33.510135 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.510242 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.510274 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:16:33.510614 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.505571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34535 (local address 127.0.98.3:56043)
0620 00:16:33.505771 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.505807 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.506326 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.506572 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.506576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.506584 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:33.507089 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.507093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.507807 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.507808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.508724 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.508726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.509177 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.509178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.509315 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.509914 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:16:33.509915 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:33.509946 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:16:33.509970 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:33.510146 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:33.510150 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:33.510154 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.510243 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.510345 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
I20260620 00:16:33.511089 20547 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34535
I20260620 00:16:33.511168 20547 heartbeater.cc:464] Registering TS with master...
I20260620 00:16:33.511370 20547 heartbeater.cc:511] Master 127.0.98.62:34535 requested a full tablet report, sending...
I20260620 00:16:33.511756 20096 ts_manager.cc:194] Registered new tserver with Master: a975b7e7f00742588f669d9bffe6e514 (127.0.98.3:42187)
I20260620 00:16:33.512099 20096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56043
I20260620 00:16:33.520665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:16:33.525035 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.522124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.0.98.62:34535)
0620 00:16:33.522196 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.522199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.522308 (+ 109us) server_negotiation.cc:407] Connection header received
0620 00:16:33.522350 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.522351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.522365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.522417 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:33.522577 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.523061 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.523829 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.524012 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.524064 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.524736 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:33.524737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:33.524737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:33.524743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:33.524758 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:33.524764 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:33.524813 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.524877 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.524901 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:16:33.526660 20096 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:16:33.528319 20096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:16:33.535859 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.530897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42359 (local address 127.0.0.1:56076)
0620 00:16:33.531040 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.531082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.531150 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.531540 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.531543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.531546 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:33.531627 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.531629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.532869 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.532871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.534528 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.534534 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.535533 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.535534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.535616 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.535630 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.535703 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.535759 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":42,"thread_start_us":96,"threads_started":1}
I20260620 00:16:33.536114 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.530903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34167 (local address 127.0.0.1:35216)
0620 00:16:33.531360 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.531371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.531415 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.532107 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.532108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.532110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:33.532182 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.532184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.533531 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.533533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.534753 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.534757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.535844 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.535845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.535918 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.535923 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.535975 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.536023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":99,"threads_started":1}
I20260620 00:16:33.536440 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.531218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.0.98.2:34167)
0620 00:16:33.531656 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.531660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.531697 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:16:33.531735 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.531737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.531752 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.531829 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:33.532271 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.533416 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.534864 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.535638 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.535717 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.535806 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.536294 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.536347 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":94,"threads_started":1}
I20260620 00:16:33.536592 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.531181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.98.1:42359)
0620 00:16:33.531337 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.531343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.531354 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:16:33.531398 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.531400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.531416 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.531508 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:33.531722 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.532777 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.534686 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.535423 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.535529 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.535643 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.536400 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.536452 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 00:16:33.537205 20350 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 (DEFAULT_TABLE table=test-table [id=fef489c5252847f088c0017d9727e82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:33.537209 20219 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 (DEFAULT_TABLE table=test-table [id=fef489c5252847f088c0017d9727e82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:33.537586 20350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:33.537586 20219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:33.541559 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.533509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42187 (local address 127.0.0.1:50748)
0620 00:16:33.535953 (+ 2444us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.535968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.536018 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.537054 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.537056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.537057 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:33.537095 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.537096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.538712 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.538714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.539928 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.539932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.541289 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.541290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.541352 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.541356 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.541426 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.541472 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2433,"thread_start_us":86,"threads_started":1}
I20260620 00:16:33.541564 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.533802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.0.98.3:42187)
0620 00:16:33.534977 (+ 1175us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.534982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.536882 (+ 1900us) server_negotiation.cc:407] Connection header received
0620 00:16:33.536935 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.536938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.536955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.537170 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:33.537234 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.538601 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.540044 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.541164 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.541274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.541369 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.541441 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.541477 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":74,"threads_started":1}
I20260620 00:16:33.542047 20561 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: Bootstrap starting.
I20260620 00:16:33.542989 20561 tablet_bootstrap.cc:654] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:33.542896 20481 tablet_service.cc:1467] Processing CreateTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 (DEFAULT_TABLE table=test-table [id=fef489c5252847f088c0017d9727e82a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:16:33.543419 20481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:33.543556 20561 log.cc:824] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:33.543815 20562 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Bootstrap starting.
I20260620 00:16:33.544615 20562 tablet_bootstrap.cc:654] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:33.544962 20562 log.cc:824] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:33.545837 20562 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: No bootstrap required, opened a new log
I20260620 00:16:33.545918 20562 ts_tablet_manager.cc:1397] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:16:33.547230 20561 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: No bootstrap required, opened a new log
I20260620 00:16:33.547251 20565 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Bootstrap starting.
I20260620 00:16:33.547307 20561 ts_tablet_manager.cc:1397] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 00:16:33.547896 20565 tablet_bootstrap.cc:654] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:33.548211 20565 log.cc:824] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:33.549180 20565 tablet_bootstrap.cc:492] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: No bootstrap required, opened a new log
I20260620 00:16:33.549263 20565 ts_tablet_manager.cc:1397] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:16:33.549463 20562 raft_consensus.cc:348] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.549733 20562 raft_consensus.cc:374] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:33.549778 20562 raft_consensus.cc:729] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe6daecd6bd43eca446e9bdac52f326, State: Initialized, Role: FOLLOWER
I20260620 00:16:33.549911 20562 consensus_queue.cc:260] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.550206 20562 ts_tablet_manager.cc:1428] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 00:16:33.550357 20416 heartbeater.cc:503] Master 127.0.98.62:34535 was elected leader, sending a full tablet report...
I20260620 00:16:33.551261 20561 raft_consensus.cc:348] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.551440 20561 raft_consensus.cc:374] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:33.551740 20561 raft_consensus.cc:729] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e44d0a667fc42cd95cdc18c27738b05, State: Initialized, Role: FOLLOWER
I20260620 00:16:33.551888 20561 consensus_queue.cc:260] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.551962 20565 raft_consensus.cc:348] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.552124 20565 raft_consensus.cc:374] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:33.552162 20565 raft_consensus.cc:729] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a975b7e7f00742588f669d9bffe6e514, State: Initialized, Role: FOLLOWER
I20260620 00:16:33.552242 20285 heartbeater.cc:503] Master 127.0.98.62:34535 was elected leader, sending a full tablet report...
I20260620 00:16:33.552268 20565 consensus_queue.cc:260] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.552524 20565 ts_tablet_manager.cc:1428] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:16:33.552574 20547 heartbeater.cc:503] Master 127.0.98.62:34535 was elected leader, sending a full tablet report...
I20260620 00:16:33.552202 20561 ts_tablet_manager.cc:1428] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
W20260620 00:16:33.621485 20417 tablet.cc:2369] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:16:33.673466 20568 raft_consensus.cc:482] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:16:33.673614 20568 raft_consensus.cc:504] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.673978 20568 leader_election.cc:288] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe6daecd6bd43eca446e9bdac52f326 (127.0.98.2:34167), a975b7e7f00742588f669d9bffe6e514 (127.0.98.3:42187)
I20260620 00:16:33.676890 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.674250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50833 (local address 127.0.98.2:34167)
0620 00:16:33.674294 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.674297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.674307 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:16:33.674342 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.674343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.674357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.674423 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:33.674612 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.675155 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676143 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.676509 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676602 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.676614 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.676766 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.676808 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:16:33.676908 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.674246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47461 (local address 127.0.98.3:42187)
0620 00:16:33.674289 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.674291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.674448 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:16:33.674477 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.674478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.674488 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.674543 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:16:33.674747 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.675212 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676186 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.676523 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676640 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.676650 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.676795 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.676835 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:16:33.676935 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.674137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34167 (local address 127.0.98.1:50833)
0620 00:16:33.674219 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.674232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.674281 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.674439 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.674440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.674442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:33.674525 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.674527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.675237 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.675238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.676039 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.676041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.676614 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.676687 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.676691 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.676741 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.676809 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":13}
I20260620 00:16:33.676947 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.674182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42187 (local address 127.0.98.1:47461)
0620 00:16:33.674340 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.674350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:33.674412 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:33.674604 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:33.674605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:33.674607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:16:33.674672 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.674674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.675302 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.675303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.676068 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:33.676070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.676641 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.676643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:33.676705 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.676708 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:16:33.676779 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:16:33.676815 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260620 00:16:33.677271 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0ed38fad0542afbc2c5542fe3ed4d7" candidate_uuid: "1e44d0a667fc42cd95cdc18c27738b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe6daecd6bd43eca446e9bdac52f326" is_pre_election: true
I20260620 00:16:33.677433 20370 raft_consensus.cc:2454] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e44d0a667fc42cd95cdc18c27738b05 in term 0.
I20260620 00:16:33.677429 20501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0ed38fad0542afbc2c5542fe3ed4d7" candidate_uuid: "1e44d0a667fc42cd95cdc18c27738b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a975b7e7f00742588f669d9bffe6e514" is_pre_election: true
I20260620 00:16:33.677596 20501 raft_consensus.cc:2454] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e44d0a667fc42cd95cdc18c27738b05 in term 0.
I20260620 00:16:33.677687 20174 leader_election.cc:302] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e44d0a667fc42cd95cdc18c27738b05, fbe6daecd6bd43eca446e9bdac52f326; no voters:
I20260620 00:16:33.677826 20568 raft_consensus.cc:2781] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:16:33.677886 20568 raft_consensus.cc:482] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:16:33.677903 20568 raft_consensus.cc:3037] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:33.678640 20568 raft_consensus.cc:504] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.678782 20568 leader_election.cc:288] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [CANDIDATE]: Term 1 election: Requested vote from peers fbe6daecd6bd43eca446e9bdac52f326 (127.0.98.2:34167), a975b7e7f00742588f669d9bffe6e514 (127.0.98.3:42187)
I20260620 00:16:33.678932 20501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0ed38fad0542afbc2c5542fe3ed4d7" candidate_uuid: "1e44d0a667fc42cd95cdc18c27738b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a975b7e7f00742588f669d9bffe6e514"
I20260620 00:16:33.678943 20370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b0ed38fad0542afbc2c5542fe3ed4d7" candidate_uuid: "1e44d0a667fc42cd95cdc18c27738b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe6daecd6bd43eca446e9bdac52f326"
I20260620 00:16:33.679031 20501 raft_consensus.cc:3037] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:33.679028 20370 raft_consensus.cc:3037] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:33.679723 20370 raft_consensus.cc:2454] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e44d0a667fc42cd95cdc18c27738b05 in term 1.
I20260620 00:16:33.679857 20501 raft_consensus.cc:2454] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e44d0a667fc42cd95cdc18c27738b05 in term 1.
I20260620 00:16:33.679886 20174 leader_election.cc:302] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e44d0a667fc42cd95cdc18c27738b05, fbe6daecd6bd43eca446e9bdac52f326; no voters:
I20260620 00:16:33.679996 20568 raft_consensus.cc:2781] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:33.680156 20568 raft_consensus.cc:686] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 1 LEADER]: Becoming Leader. State: Replica: 1e44d0a667fc42cd95cdc18c27738b05, State: Running, Role: LEADER
I20260620 00:16:33.680323 20568 consensus_queue.cc:237] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } }
I20260620 00:16:33.681361 20096 catalog_manager.cc:5495] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e44d0a667fc42cd95cdc18c27738b05 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1e44d0a667fc42cd95cdc18c27738b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e44d0a667fc42cd95cdc18c27738b05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } health_report { overall_health: UNKNOWN } } }
I20260620 00:16:33.692142 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.689665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.0.98.1:42359)
0620 00:16:33.689730 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:33.689733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:33.689742 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:33.689820 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:33.689821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:33.689843 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:33.689929 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:16:33.690091 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.690576 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.691316 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:33.691528 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:33.691608 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:33.691770 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:16:33.691884 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:16:33.691979 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:33.692004 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:16:33.692023 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:16:33.694136 20501 raft_consensus.cc:1261] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 1 FOLLOWER]: Refusing update from remote peer 1e44d0a667fc42cd95cdc18c27738b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:33.694136 20370 raft_consensus.cc:1261] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 1 FOLLOWER]: Refusing update from remote peer 1e44d0a667fc42cd95cdc18c27738b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:33.694536 20571 consensus_queue.cc:1035] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a975b7e7f00742588f669d9bffe6e514" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:33.694680 20568 consensus_queue.cc:1035] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe6daecd6bd43eca446e9bdac52f326" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:33.697702 20576 mvcc.cc:205] Tried to move back new op lower bound from 7298722175768215552 to 7298722175715205120. Current Snapshot: MvccSnapshot[applied={T|T < 7298722175768215552}]
I20260620 00:16:33.700034 20097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:33.700125 20097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:16:33.700963 20097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98d5a720a5c244ce862401d32c0affc9: Sending DeleteTablet for 3 replicas of tablet 7b0ed38fad0542afbc2c5542fe3ed4d7
I20260620 00:16:33.701333 20350 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:33 UTC) from {username='slave'} at 127.0.0.1:35216
I20260620 00:16:33.701375 20219 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:33 UTC) from {username='slave'} at 127.0.0.1:56076
I20260620 00:16:33.701606 20481 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:16:33 UTC) from {username='slave'} at 127.0.0.1:50748
I20260620 00:16:33.701650 20585 tablet_replica.cc:330] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05: stopping tablet replica
I20260620 00:16:33.701818 20584 tablet_replica.cc:330] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: stopping tablet replica
I20260620 00:16:33.701829 20585 raft_consensus.cc:2229] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:16:33.701929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20157
I20260620 00:16:33.701927 20584 raft_consensus.cc:2229] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:33.702085 20584 raft_consensus.cc:2258] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:33.702100 20585 raft_consensus.cc:2258] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P 1e44d0a667fc42cd95cdc18c27738b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:33.702239 20586 tablet_replica.cc:330] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: stopping tablet replica
I20260620 00:16:33.702531 20586 raft_consensus.cc:2229] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:16:33.702751 20586 raft_consensus.cc:2258] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:16:33.703332 20584 ts_tablet_manager.cc:1905] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:33.703332 20586 ts_tablet_manager.cc:1905] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:16:33.705551 20584 ts_tablet_manager.cc:1918] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:33.705578 20586 ts_tablet_manager.cc:1918] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:16:33.705649 20584 log.cc:1196] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-1/wal/wals/7b0ed38fad0542afbc2c5542fe3ed4d7
I20260620 00:16:33.705649 20586 log.cc:1196] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Deleting WAL directory at /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914322045185-392-0/minicluster-data/ts-2/wal/wals/7b0ed38fad0542afbc2c5542fe3ed4d7
I20260620 00:16:33.705920 20584 ts_tablet_manager.cc:1939] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P fbe6daecd6bd43eca446e9bdac52f326: Deleting consensus metadata
I20260620 00:16:33.705922 20586 ts_tablet_manager.cc:1939] T 7b0ed38fad0542afbc2c5542fe3ed4d7 P a975b7e7f00742588f669d9bffe6e514: Deleting consensus metadata
I20260620 00:16:33.706411 20083 catalog_manager.cc:4841] TS fbe6daecd6bd43eca446e9bdac52f326 (127.0.98.2:34167): tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 (table test-table [id=fef489c5252847f088c0017d9727e82a]) successfully deleted
I20260620 00:16:33.706908 20082 catalog_manager.cc:4841] TS a975b7e7f00742588f669d9bffe6e514 (127.0.98.3:42187): tablet 7b0ed38fad0542afbc2c5542fe3ed4d7 (table test-table [id=fef489c5252847f088c0017d9727e82a]) successfully deleted
W20260620 00:16:33.708679 20081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42359 (error 108)
I20260620 00:16:33.708849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20288
I20260620 00:16:33.709599 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:33.709457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42359 (local address 127.0.0.1:56098)
0620 00:16:33.709518 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:33.709571 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 00:16:33.709719 20081 catalog_manager.cc:4584] TS 1e44d0a667fc42cd95cdc18c27738b05 (127.0.98.1:42359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b0ed38fad0542afbc2c5542fe3ed4d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42359: connect: Connection refused (error 111)
I20260620 00:16:33.714680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20419
I20260620 00:16:33.719147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20066
2026-06-20T00:16:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (795 ms)
[----------] 6 tests from EncryptionPolicyTest (8896 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:33 dist-test-slave-28bv krb5kdc[20590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:33 dist-test-slave-28bv krb5kdc[20590](info): set up 2 sockets
Jun 20 00:16:33 dist-test-slave-28bv krb5kdc[20590](info): commencing operation
krb5kdc: starting...
W20260620 00:16:34.768991 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:34 dist-test-slave-28bv krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:34.895653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:34.978425 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:34.978664 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:34.978688 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:34.979830 20603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:34.979871 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:34.979884 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:34.979897 20603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:34.979909 20603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:34.981496 20603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:34.981709 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:34.981891 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:16:34.986981 20603 server_base.cc:1034] running on GCE node
W20260620 00:16:34.986974 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:34.986974 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:34.987006 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:34.987640 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:34.987958 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:34.989164 20603 hybrid_clock.cc:648] HybridClock initialized: now 1781914594989135 us; error 40 us; skew 500 ppm
Jun 20 00:16:34 dist-test-slave-28bv krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:34.995501 20603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:34.995944 20603 webserver.cc:460] Webserver started at http://127.0.0.1:42111/ using document root <none> and password file <none>
I20260620 00:16:34.996213 20603 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:34.996284 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:34.996449 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:34.997828 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7b8473a5964341b5bb36f5bb71b7bdf6"
format_stamp: "Formatted at 2026-06-20 00:16:34 on dist-test-slave-28bv"
server_key: "735306804ff3f5ff2a657800d669f6b4"
server_key_iv: "fc2674caec72685099c5a12b42bafb8e"
server_key_version: "encryptionkey@0"
I20260620 00:16:34.998221 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b8473a5964341b5bb36f5bb71b7bdf6"
format_stamp: "Formatted at 2026-06-20 00:16:34 on dist-test-slave-28bv"
server_key: "735306804ff3f5ff2a657800d669f6b4"
server_key_iv: "fc2674caec72685099c5a12b42bafb8e"
server_key_version: "encryptionkey@0"
I20260620 00:16:34.999686 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:35.000751 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:35.000972 20603 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:35.001102 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7b8473a5964341b5bb36f5bb71b7bdf6"
format_stamp: "Formatted at 2026-06-20 00:16:34 on dist-test-slave-28bv"
server_key: "735306804ff3f5ff2a657800d669f6b4"
server_key_iv: "fc2674caec72685099c5a12b42bafb8e"
server_key_version: "encryptionkey@0"
I20260620 00:16:35.001199 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:35.009156 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:35.011377 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:35.011561 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:35.015870 20603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43167
I20260620 00:16:35.015923 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43167 every 8 connection(s)
I20260620 00:16:35.016312 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:35.017335 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:35.019660 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6: Bootstrap starting.
I20260620 00:16:35.020226 20671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:35.020465 20671 log.cc:824] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:35.020807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20603
I20260620 00:16:35.020943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:35.021191 392 external_mini_cluster.cc:1428] Setting key 59792caa65d9dfd5004f522afc43dc9e
I20260620 00:16:35.021459 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6: No bootstrap required, opened a new log
I20260620 00:16:35.023537 20671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260620 00:16:35.023728 20671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:35.023788 20671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b8473a5964341b5bb36f5bb71b7bdf6, State: Initialized, Role: FOLLOWER
I20260620 00:16:35.023958 20671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260620 00:16:35.024060 20671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:35.024124 20671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:35.024190 20671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:35.025274 20671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260620 00:16:35.025410 20671 leader_election.cc:302] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b8473a5964341b5bb36f5bb71b7bdf6; no voters:
I20260620 00:16:35.025624 20671 leader_election.cc:288] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:35.025676 20676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:35.025815 20676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [term 1 LEADER]: Becoming Leader. State: Replica: 7b8473a5964341b5bb36f5bb71b7bdf6, State: Running, Role: LEADER
I20260620 00:16:35.025887 20671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:35.025897 20676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260620 00:16:35.026432 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b8473a5964341b5bb36f5bb71b7bdf6. Latest consensus state: current_term: 1 leader_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260620 00:16:35.026520 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:35.026683 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b8473a5964341b5bb36f5bb71b7bdf6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260620 00:16:35.026736 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:35.026829 20684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:35.027329 20684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:16:35 dist-test-slave-28bv krb5kdc[20590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:16:35.031402 20684 catalog_manager.cc:1347] Generated new cluster ID: 44e107a5c22a412db9a76284271435db
I20260620 00:16:35.031464 20684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:35.035199 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:35.021857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58635 (local address 127.0.0.1:43167)
0620 00:16:35.022033 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:35.022042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:35.022113 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:16:35.023527 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:35.023535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:35.023592 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:35.023720 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:35.024756 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:35.025945 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:35.027085 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:35.027719 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:35.028215 (+ 496us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:35.031087 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:35.031091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:35.031092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:35.031104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:35.033332 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:35.033828 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:35.033831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:35.033832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:35.033983 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:35.034218 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:35.034219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:35.034220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:35.034571 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:35.034740 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:35.034862 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 00:16:35.034982 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 00:16:35.036552 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:35.038828 20684 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:35.039347 20684 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:35.043159 20684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b8473a5964341b5bb36f5bb71b7bdf6: Generated new TSK 0
I20260620 00:16:35.043372 20684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:35.050521 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:35.038623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35765 (local address 127.0.0.1:43167)
0620 00:16:35.038699 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:35.038701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:35.042956 (+ 4255us) server_negotiation.cc:407] Connection header received
0620 00:16:35.043058 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:35.043059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:35.043071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:35.043142 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:35.043887 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:35.044437 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:35.045225 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:35.045464 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:35.045555 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:35.046956 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:35.046958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:35.046958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:35.046965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:35.049142 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:35.049623 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:35.049625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:35.049625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:35.049686 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:35.049929 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:35.049930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:35.049931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:35.050053 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:35.050172 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:35.050300 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 00:16:35.050403 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:16:35.052903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20603
W20260620 00:16:35.057894 20588 connection.cc:535] client connection to 127.0.0.1:43167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43167): Connection reset by peer (error 104)
2026-06-20T00:16:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:35 dist-test-slave-28bv krb5kdc[20702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:35 dist-test-slave-28bv krb5kdc[20702](info): set up 2 sockets
Jun 20 00:16:35 dist-test-slave-28bv krb5kdc[20702](info): commencing operation
krb5kdc: starting...
W20260620 00:16:36.109225 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:36 dist-test-slave-28bv krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:36.257063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:36.346038 20715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:36.346321 20715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:36.346354 20715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:36.347913 20715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:36.347982 20715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:36.347999 20715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:36.348014 20715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:36.348028 20715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:36.350100 20715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:36.350466 20715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:36.350745 20715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:16:36.357219 20715 server_base.cc:1034] running on GCE node
W20260620 00:16:36.357035 20720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:36.357007 20723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:36.356976 20721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:36.357910 20715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:36.358281 20715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:36.359709 20715 hybrid_clock.cc:648] HybridClock initialized: now 1781914596359671 us; error 79 us; skew 500 ppm
Jun 20 00:16:36 dist-test-slave-28bv krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:36.362766 20715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:36.363322 20715 webserver.cc:460] Webserver started at http://127.0.0.1:37097/ using document root <none> and password file <none>
I20260620 00:16:36.363598 20715 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:36.363649 20715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:36.363847 20715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:36.365229 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "4cd204f2283d470997ed16fc7454df87"
format_stamp: "Formatted at 2026-06-20 00:16:36 on dist-test-slave-28bv"
server_key: "73def5090855acfb057ac90618fd869c"
server_key_iv: "6d2b90d90de3d550b013d50002fc0dd6"
server_key_version: "encryptionkey@0"
I20260620 00:16:36.365657 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "4cd204f2283d470997ed16fc7454df87"
format_stamp: "Formatted at 2026-06-20 00:16:36 on dist-test-slave-28bv"
server_key: "73def5090855acfb057ac90618fd869c"
server_key_iv: "6d2b90d90de3d550b013d50002fc0dd6"
server_key_version: "encryptionkey@0"
I20260620 00:16:36.367350 20715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:16:36.368718 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:36.369031 20715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:36.369202 20715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "4cd204f2283d470997ed16fc7454df87"
format_stamp: "Formatted at 2026-06-20 00:16:36 on dist-test-slave-28bv"
server_key: "73def5090855acfb057ac90618fd869c"
server_key_iv: "6d2b90d90de3d550b013d50002fc0dd6"
server_key_version: "encryptionkey@0"
I20260620 00:16:36.369313 20715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:36.383203 20715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:36.385795 20715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:36.385983 20715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:36.390607 20715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36911
I20260620 00:16:36.390659 20782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36911 every 8 connection(s)
I20260620 00:16:36.391048 20715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:36.391980 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:36.393187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20715
I20260620 00:16:36.393292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:36.393517 392 external_mini_cluster.cc:1428] Setting key 59f4df23227f86d12f50e32c32d7acb6
I20260620 00:16:36.395159 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87: Bootstrap starting.
I20260620 00:16:36.395851 20783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:36.396169 20783 log.cc:824] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:36.397032 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87: No bootstrap required, opened a new log
I20260620 00:16:36.399598 20783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260620 00:16:36.399778 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:36.394114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34431 (local address 127.0.0.1:36911)
0620 00:16:36.394292 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:36.394300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:36.394328 (+ 28us) server_negotiation.cc:407] Connection header received
0620 00:16:36.394930 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:36.394938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:36.394992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:36.395129 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:36.395389 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:36.396445 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:36.397385 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:36.398030 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:36.398147 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:36.398972 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:36.398978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:36.398978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:36.399243 (+ 265us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:36.399289 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:36.399308 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:36.399398 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:36.399479 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 00:16:36.399520 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260620 00:16:36.399806 20783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:36.399842 20783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cd204f2283d470997ed16fc7454df87, State: Initialized, Role: FOLLOWER
I20260620 00:16:36.399959 20783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260620 00:16:36.400053 20783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:36.400097 20783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:36.400148 20783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:36.400977 20783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260620 00:16:36.401094 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:36.401130 20783 leader_election.cc:302] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cd204f2283d470997ed16fc7454df87; no voters:
I20260620 00:16:36.401333 20783 leader_election.cc:288] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:36.401391 20788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:36.401572 20788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [term 1 LEADER]: Becoming Leader. State: Replica: 4cd204f2283d470997ed16fc7454df87, State: Running, Role: LEADER
I20260620 00:16:36.401638 20783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:36.401683 20788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } }
I20260620 00:16:36.402081 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cd204f2283d470997ed16fc7454df87. Latest consensus state: current_term: 1 leader_uuid: "4cd204f2283d470997ed16fc7454df87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } } }
I20260620 00:16:36.402096 20789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cd204f2283d470997ed16fc7454df87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cd204f2283d470997ed16fc7454df87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36911 } } }
I20260620 00:16:36.402172 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:36.402226 20789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:36.403337 20807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:36.404129 20807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:36.405848 20807 catalog_manager.cc:1347] Generated new cluster ID: 51351d38ab9a4e3ab957c57984751f97
I20260620 00:16:36.405915 20807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:36.407542 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:36.403496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60499 (local address 127.0.0.1:36911)
0620 00:16:36.403549 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:36.403552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:36.403708 (+ 156us) server_negotiation.cc:407] Connection header received
0620 00:16:36.403736 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:36.403737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:36.403765 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:36.403821 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:36.404012 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:36.404759 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:36.406071 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:36.406387 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:36.406461 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:36.407272 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:36.407273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:36.407274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:36.407280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:36.407296 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:36.407301 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:36.407356 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:36.407454 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:16:36.407466 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:16:36.414477 20807 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:36.415060 20807 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:36.420811 20807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cd204f2283d470997ed16fc7454df87: Generated new TSK 0
I20260620 00:16:36.421052 20807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:36.424445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20715
W20260620 00:16:36.428579 20700 connection.cc:535] client connection to 127.0.0.1:36911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36911): Connection reset by peer (error 104)
2026-06-20T00:16:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:36 dist-test-slave-28bv krb5kdc[20814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:36 dist-test-slave-28bv krb5kdc[20814](info): set up 2 sockets
Jun 20 00:16:36 dist-test-slave-28bv krb5kdc[20814](info): commencing operation
krb5kdc: starting...
W20260620 00:16:37.496764 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:37 dist-test-slave-28bv krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:37.620194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:37.702056 20827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:37.702347 20827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:37.702389 20827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:37.703640 20827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:37.703727 20827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:37.703752 20827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:37.703773 20827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:37.703830 20827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:37.705466 20827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:37.705803 20827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:37.706063 20827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:37.711069 20835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:37.711032 20833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:37.711210 20827 server_base.cc:1034] running on GCE node
W20260620 00:16:37.711032 20832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:37.711674 20827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:37.711989 20827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:37.713160 20827 hybrid_clock.cc:648] HybridClock initialized: now 1781914597713152 us; error 37 us; skew 500 ppm
Jun 20 00:16:37 dist-test-slave-28bv krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:37.715683 20827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:37.716154 20827 webserver.cc:460] Webserver started at http://127.0.0.1:44273/ using document root <none> and password file <none>
I20260620 00:16:37.716409 20827 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:37.716495 20827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:37.716661 20827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:37.717751 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "891d48efb75049699858891baaf11214"
format_stamp: "Formatted at 2026-06-20 00:16:37 on dist-test-slave-28bv"
server_key: "101b249ad455774d0208606bb4e0d849"
server_key_iv: "ee53b84b4f4cf364f1fc3ac5b863e125"
server_key_version: "encryptionkey@0"
I20260620 00:16:37.718128 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "891d48efb75049699858891baaf11214"
format_stamp: "Formatted at 2026-06-20 00:16:37 on dist-test-slave-28bv"
server_key: "101b249ad455774d0208606bb4e0d849"
server_key_iv: "ee53b84b4f4cf364f1fc3ac5b863e125"
server_key_version: "encryptionkey@0"
I20260620 00:16:37.719574 20827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:37.720456 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:37.720669 20827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:37.720784 20827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "891d48efb75049699858891baaf11214"
format_stamp: "Formatted at 2026-06-20 00:16:37 on dist-test-slave-28bv"
server_key: "101b249ad455774d0208606bb4e0d849"
server_key_iv: "ee53b84b4f4cf364f1fc3ac5b863e125"
server_key_version: "encryptionkey@0"
I20260620 00:16:37.720866 20827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:37.724262 20827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:37.726501 20827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:37.726670 20827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:37.730924 20827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42509
I20260620 00:16:37.730978 20894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42509 every 8 connection(s)
I20260620 00:16:37.731351 20827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:37.732395 20895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:37.734745 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: Bootstrap starting.
I20260620 00:16:37.735283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20827
I20260620 00:16:37.735406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:37.735476 20895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:37.735632 392 external_mini_cluster.cc:1428] Setting key 3a310eb0fe7f5d6728224a419ecaf263
I20260620 00:16:37.735810 20895 log.cc:824] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:37.736722 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: No bootstrap required, opened a new log
I20260620 00:16:37.738337 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:37.736163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47603 (local address 127.0.0.1:42509)
0620 00:16:37.736329 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:37.736338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:37.736376 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:16:37.736958 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:37.736965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:37.736992 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:37.737117 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:37.737832 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:37.737836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:37.737836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:37.737840 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:37.737881 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:37.737892 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:37.737975 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:37.738051 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:16:37.738117 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260620 00:16:37.739316 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:37.739257 20895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260620 00:16:37.739454 20895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:37.739494 20895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891d48efb75049699858891baaf11214, State: Initialized, Role: FOLLOWER
I20260620 00:16:37.739611 20895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260620 00:16:37.739691 20895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:37.739718 20895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:37.739765 20895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:37.740636 20895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260620 00:16:37.740761 20895 leader_election.cc:302] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891d48efb75049699858891baaf11214; no voters:
I20260620 00:16:37.740959 20895 leader_election.cc:288] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:37.740998 20904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:37.741190 20904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [term 1 LEADER]: Becoming Leader. State: Replica: 891d48efb75049699858891baaf11214, State: Running, Role: LEADER
I20260620 00:16:37.741245 20895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:37.741288 20904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } }
I20260620 00:16:37.741653 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891d48efb75049699858891baaf11214. Latest consensus state: current_term: 1 leader_uuid: "891d48efb75049699858891baaf11214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } } }
I20260620 00:16:37.741792 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:37.741976 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891d48efb75049699858891baaf11214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891d48efb75049699858891baaf11214" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42509 } } }
I20260620 00:16:37.742096 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:37.742395 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:37.741015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53039 (local address 127.0.0.1:42509)
0620 00:16:37.741051 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:37.741054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:37.741159 (+ 105us) server_negotiation.cc:407] Connection header received
0620 00:16:37.741253 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:37.741255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:37.741260 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:37.741290 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:37.742030 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:37.742032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:37.742033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:37.742039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:37.742056 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:37.742061 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:37.742104 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:37.742295 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 00:16:37.742307 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 00:16:37.743165 20920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:37.743221 20920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:37.743265 20921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:37.743469 20921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:37.745095 20921 catalog_manager.cc:1347] Generated new cluster ID: 41d9b96bae5f4adb9ba155c79848259e
I20260620 00:16:37.745157 20921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:37.758579 20921 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:37.758783 20921 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:37.762888 20921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891d48efb75049699858891baaf11214: Generated new TSK 0
I20260620 00:16:37.763182 20921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:37.771010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20827
2026-06-20T00:16:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:37 dist-test-slave-28bv krb5kdc[20926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:37 dist-test-slave-28bv krb5kdc[20926](info): set up 2 sockets
Jun 20 00:16:37 dist-test-slave-28bv krb5kdc[20926](info): commencing operation
krb5kdc: starting...
W20260620 00:16:38.808699 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:38 dist-test-slave-28bv krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:38.936790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:39.019093 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:39.019392 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:39.019434 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:39.020803 20939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:39.020879 20939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:39.020905 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:39.020933 20939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:39.020965 20939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:39.022547 20939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:39.022855 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:39.023102 20939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:39.028247 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:39.028229 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:39.028262 20939 server_base.cc:1034] running on GCE node
W20260620 00:16:39.028231 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:39.028950 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:39.029294 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:39.030490 20939 hybrid_clock.cc:648] HybridClock initialized: now 1781914599030466 us; error 31 us; skew 500 ppm
Jun 20 00:16:39 dist-test-slave-28bv krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:39.033246 20939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:39.033746 20939 webserver.cc:460] Webserver started at http://127.0.0.1:33065/ using document root <none> and password file <none>
I20260620 00:16:39.034000 20939 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:39.034070 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:39.034219 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:39.035311 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7c316b8bb1944b43a404d90f5a7b9d2c"
format_stamp: "Formatted at 2026-06-20 00:16:39 on dist-test-slave-28bv"
server_key: "43623fee92ab403842219386246c8315"
server_key_iv: "e76fe9541e59a42221703e55a312b3c1"
server_key_version: "encryptionkey@0"
I20260620 00:16:39.035686 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c316b8bb1944b43a404d90f5a7b9d2c"
format_stamp: "Formatted at 2026-06-20 00:16:39 on dist-test-slave-28bv"
server_key: "43623fee92ab403842219386246c8315"
server_key_iv: "e76fe9541e59a42221703e55a312b3c1"
server_key_version: "encryptionkey@0"
I20260620 00:16:39.037122 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:39.038010 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:39.038203 20939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:39.038296 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7c316b8bb1944b43a404d90f5a7b9d2c"
format_stamp: "Formatted at 2026-06-20 00:16:39 on dist-test-slave-28bv"
server_key: "43623fee92ab403842219386246c8315"
server_key_iv: "e76fe9541e59a42221703e55a312b3c1"
server_key_version: "encryptionkey@0"
I20260620 00:16:39.038379 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:39.049638 20939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:39.051852 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:39.052037 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:39.056499 20939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41725
I20260620 00:16:39.056622 21006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41725 every 8 connection(s)
I20260620 00:16:39.057005 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:39.057893 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:39.060171 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c: Bootstrap starting.
I20260620 00:16:39.060849 21007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:39.061141 21007 log.cc:824] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:39.061828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 20939
I20260620 00:16:39.061965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:39.062027 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c: No bootstrap required, opened a new log
I20260620 00:16:39.062179 392 external_mini_cluster.cc:1428] Setting key 694815c4b8816a12680bb9ac0e46a93f
I20260620 00:16:39.064589 21007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260620 00:16:39.064750 21007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:39.064775 21007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c316b8bb1944b43a404d90f5a7b9d2c, State: Initialized, Role: FOLLOWER
I20260620 00:16:39.064913 21007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260620 00:16:39.064994 21007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:39.065037 21007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:39.065086 21007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:39.065743 21007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260620 00:16:39.065830 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c316b8bb1944b43a404d90f5a7b9d2c; no voters:
I20260620 00:16:39.066082 21007 leader_election.cc:288] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:39.066107 21012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:39.066223 21012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [term 1 LEADER]: Becoming Leader. State: Replica: 7c316b8bb1944b43a404d90f5a7b9d2c, State: Running, Role: LEADER
I20260620 00:16:39.066352 21007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:39.066367 21012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } }
I20260620 00:16:39.066797 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c316b8bb1944b43a404d90f5a7b9d2c. Latest consensus state: current_term: 1 leader_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260620 00:16:39.067030 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:39.066957 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c316b8bb1944b43a404d90f5a7b9d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41725 } } }
I20260620 00:16:39.067261 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:39.067437 21021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:39.067955 21021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:39.069581 21021 catalog_manager.cc:1347] Generated new cluster ID: c045fcb826ce4cf1a3b2f893d0019923
I20260620 00:16:39.069638 21021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:16:39 dist-test-slave-28bv krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:16:39.074818 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:39.062708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.0.0.1:41725)
0620 00:16:39.062917 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:39.062929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:39.062960 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:16:39.063454 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:39.063462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:39.063512 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:39.063636 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:39.064484 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:39.065897 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:39.066998 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:39.067632 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:39.067723 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:39.070852 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:39.070856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:39.070856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:39.070868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:39.072979 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:39.073399 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:39.073402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:39.073403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:39.073512 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:39.073721 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:39.073722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:39.073722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:39.074044 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:39.074201 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:39.074314 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 00:16:39.074467 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":119,"threads_started":1}
I20260620 00:16:39.075789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:39.084350 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:39.077324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51823 (local address 127.0.0.1:41725)
0620 00:16:39.077366 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:39.077368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:39.077399 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:16:39.077418 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:39.077419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:39.077428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:39.077479 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:39.078076 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:39.078830 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:39.079814 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:39.080130 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:39.080204 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:39.081600 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:39.081601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:39.081601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:39.081609 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:39.083187 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:39.083569 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:39.083570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:39.083570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:39.083632 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:39.083825 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:39.083826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:39.083826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:39.083959 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:39.084057 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:39.084163 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:16:39.084247 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 00:16:39.089303 21021 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:39.090077 21021 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:39.094525 21021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c316b8bb1944b43a404d90f5a7b9d2c: Generated new TSK 0
I20260620 00:16:39.094723 21021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:39.102566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 20939
2026-06-20T00:16:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:39 dist-test-slave-28bv krb5kdc[21038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:39 dist-test-slave-28bv krb5kdc[21038](info): set up 2 sockets
Jun 20 00:16:39 dist-test-slave-28bv krb5kdc[21038](info): commencing operation
krb5kdc: starting...
W20260620 00:16:40.144800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:40 dist-test-slave-28bv krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:40.276139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:40.359385 21051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:40.359625 21051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:40.359651 21051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:40.360844 21051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:40.360888 21051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:40.360903 21051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:40.360914 21051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:40.360926 21051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:40.362458 21051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:40.362680 21051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:40.362883 21051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:40.368065 21056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:40.368072 21059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:40.368072 21057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:40.368248 21051 server_base.cc:1034] running on GCE node
I20260620 00:16:40.368729 21051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:40.369047 21051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:40.370189 21051 hybrid_clock.cc:648] HybridClock initialized: now 1781914600370157 us; error 40 us; skew 500 ppm
Jun 20 00:16:40 dist-test-slave-28bv krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:40.377942 21051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:40.378448 21051 webserver.cc:460] Webserver started at http://127.0.0.1:42161/ using document root <none> and password file <none>
I20260620 00:16:40.378690 21051 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:40.378760 21051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:40.378928 21051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:40.380059 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "9721b851f1f64c7f9159992baef39025"
format_stamp: "Formatted at 2026-06-20 00:16:40 on dist-test-slave-28bv"
server_key: "3f18a0c411b662f0ffb03f22196e6028"
server_key_iv: "0ec9c3b395cdf86b2f58eea7ba3018a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:40.380448 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "9721b851f1f64c7f9159992baef39025"
format_stamp: "Formatted at 2026-06-20 00:16:40 on dist-test-slave-28bv"
server_key: "3f18a0c411b662f0ffb03f22196e6028"
server_key_iv: "0ec9c3b395cdf86b2f58eea7ba3018a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:40.381968 21051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:40.382736 21066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:40.382918 21051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:40.383021 21051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "9721b851f1f64c7f9159992baef39025"
format_stamp: "Formatted at 2026-06-20 00:16:40 on dist-test-slave-28bv"
server_key: "3f18a0c411b662f0ffb03f22196e6028"
server_key_iv: "0ec9c3b395cdf86b2f58eea7ba3018a7"
server_key_version: "encryptionkey@0"
I20260620 00:16:40.383118 21051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:40.396337 21051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:40.398634 21051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:40.398808 21051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:40.402959 21051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37261
I20260620 00:16:40.402992 21118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37261 every 8 connection(s)
I20260620 00:16:40.403359 21051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:40.404335 21119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:40.406755 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025: Bootstrap starting.
I20260620 00:16:40.407373 21119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:40.407666 21119 log.cc:824] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:40.408524 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025: No bootstrap required, opened a new log
I20260620 00:16:40.410239 21119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 00:16:40.410384 21119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:40.410406 21119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9721b851f1f64c7f9159992baef39025, State: Initialized, Role: FOLLOWER
I20260620 00:16:40.410573 21119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 00:16:40.410647 21119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:40.410694 21119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:40.410748 21119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:40.411173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21051
I20260620 00:16:40.411273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:40.411383 21119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 00:16:40.411506 392 external_mini_cluster.cc:1428] Setting key 15328aee3b9c48dad59a150833444a02
I20260620 00:16:40.411511 21119 leader_election.cc:302] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9721b851f1f64c7f9159992baef39025; no voters:
I20260620 00:16:40.411720 21119 leader_election.cc:288] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:40.411790 21122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:40.411984 21122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [term 1 LEADER]: Becoming Leader. State: Replica: 9721b851f1f64c7f9159992baef39025, State: Running, Role: LEADER
I20260620 00:16:40.412005 21119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:40.412174 21122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } }
I20260620 00:16:40.412664 21123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9721b851f1f64c7f9159992baef39025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260620 00:16:40.412744 21123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:40.412942 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9721b851f1f64c7f9159992baef39025. Latest consensus state: current_term: 1 leader_uuid: "9721b851f1f64c7f9159992baef39025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9721b851f1f64c7f9159992baef39025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37261 } } }
I20260620 00:16:40.413229 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:40.413278 21128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:40.413820 21128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:40.415534 21128 catalog_manager.cc:1347] Generated new cluster ID: 4edac09e21a342a3be557f21a50d0b1e
I20260620 00:16:40.415589 21128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:40.418589 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:40.412170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47901 (local address 127.0.0.1:37261)
0620 00:16:40.412441 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:40.412451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:40.412494 (+ 43us) server_negotiation.cc:407] Connection header received
0620 00:16:40.412657 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:40.412664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:40.412731 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:40.413179 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:40.413294 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:40.414765 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:40.415914 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:40.416589 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:40.416707 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:40.417609 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:40.417614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:40.417614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:40.417860 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:40.417901 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:40.417916 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:40.417977 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:40.418282 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:16:40.418354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260620 00:16:40.419708 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:40.426599 21128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:40.427314 21128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:40.428654 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:40.424913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51649 (local address 127.0.0.1:37261)
0620 00:16:40.424965 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:40.424968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:40.425151 (+ 183us) server_negotiation.cc:407] Connection header received
0620 00:16:40.425204 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:40.425205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:40.425217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:40.425281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:40.425445 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:40.426227 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:40.427248 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:40.427565 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:40.427642 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:40.428365 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:40.428366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:40.428367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:40.428374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:40.428390 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:40.428395 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:40.428455 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:40.428512 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:16:40.428575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:16:40.434638 21128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9721b851f1f64c7f9159992baef39025: Generated new TSK 0
I20260620 00:16:40.434867 21128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:40.438308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21051
2026-06-20T00:16:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:40 dist-test-slave-28bv krb5kdc[21150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:40 dist-test-slave-28bv krb5kdc[21150](info): set up 2 sockets
Jun 20 00:16:40 dist-test-slave-28bv krb5kdc[21150](info): commencing operation
krb5kdc: starting...
W20260620 00:16:41.549005 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:41 dist-test-slave-28bv krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:41.679664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36773
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:41.762002 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:41.762228 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:41.762252 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:41.763383 21163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:41.763419 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:41.763433 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:41.763446 21163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:41.763458 21163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:41.764910 21163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:41.765118 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:41.765313 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:41.770300 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:41.770345 21163 server_base.cc:1034] running on GCE node
W20260620 00:16:41.770262 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:41.770262 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:41.770740 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:41.771078 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:41.772269 21163 hybrid_clock.cc:648] HybridClock initialized: now 1781914601772217 us; error 62 us; skew 500 ppm
Jun 20 00:16:41 dist-test-slave-28bv krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:41.774951 21163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:41.775437 21163 webserver.cc:460] Webserver started at http://127.0.0.1:45309/ using document root <none> and password file <none>
I20260620 00:16:41.775677 21163 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:41.775722 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:41.775895 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:41.777089 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "c876998869644d32981729b2396a7954"
format_stamp: "Formatted at 2026-06-20 00:16:41 on dist-test-slave-28bv"
server_key: "75727c12f92470edbc5f2f0bce3baf93"
server_key_iv: "632066071a8283c810ca554f7eb1dc8d"
server_key_version: "encryptionkey@0"
I20260620 00:16:41.777451 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "c876998869644d32981729b2396a7954"
format_stamp: "Formatted at 2026-06-20 00:16:41 on dist-test-slave-28bv"
server_key: "75727c12f92470edbc5f2f0bce3baf93"
server_key_iv: "632066071a8283c810ca554f7eb1dc8d"
server_key_version: "encryptionkey@0"
I20260620 00:16:41.778894 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:16:41.779771 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:41.780009 21163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:41.780123 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "c876998869644d32981729b2396a7954"
format_stamp: "Formatted at 2026-06-20 00:16:41 on dist-test-slave-28bv"
server_key: "75727c12f92470edbc5f2f0bce3baf93"
server_key_iv: "632066071a8283c810ca554f7eb1dc8d"
server_key_version: "encryptionkey@0"
I20260620 00:16:41.780212 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:41.783739 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:41.786249 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:41.786428 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:41.790730 21163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36773
I20260620 00:16:41.790766 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36773 every 8 connection(s)
I20260620 00:16:41.791128 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:41.792198 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:41.794581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21163
I20260620 00:16:41.794695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:41.794742 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954: Bootstrap starting.
I20260620 00:16:41.794930 392 external_mini_cluster.cc:1428] Setting key 5f585638d30e5ac796750521e41185b9
I20260620 00:16:41.795524 21231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:41.795851 21231 log.cc:824] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:41.796703 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954: No bootstrap required, opened a new log
I20260620 00:16:41.797801 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:41.795476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49049 (local address 127.0.0.1:36773)
0620 00:16:41.795729 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:41.795743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:41.795770 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:16:41.796434 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:41.796441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:41.796469 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:41.796690 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:41.797351 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:41.797353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:41.797354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:41.797357 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:41.797394 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:41.797404 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:41.797476 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:41.797536 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:16:41.797591 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260620 00:16:41.798786 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:41.799399 21231 raft_consensus.cc:348] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } }
I20260620 00:16:41.799561 21231 raft_consensus.cc:374] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:41.799585 21231 raft_consensus.cc:729] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c876998869644d32981729b2396a7954, State: Initialized, Role: FOLLOWER
I20260620 00:16:41.799728 21231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } }
I20260620 00:16:41.799842 21231 raft_consensus.cc:388] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:41.799883 21231 raft_consensus.cc:482] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:41.799935 21231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:41.800803 21231 raft_consensus.cc:504] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } }
I20260620 00:16:41.800933 21231 leader_election.cc:302] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c876998869644d32981729b2396a7954; no voters:
I20260620 00:16:41.801136 21231 leader_election.cc:288] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:41.801180 21241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:41.801373 21241 raft_consensus.cc:686] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [term 1 LEADER]: Becoming Leader. State: Replica: c876998869644d32981729b2396a7954, State: Running, Role: LEADER
I20260620 00:16:41.801437 21231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:41.801568 21241 consensus_queue.cc:237] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } }
I20260620 00:16:41.801651 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:41.800350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44871 (local address 127.0.0.1:36773)
0620 00:16:41.800387 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:41.800389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:41.800497 (+ 108us) server_negotiation.cc:407] Connection header received
0620 00:16:41.800581 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:41.800582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:41.800587 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:41.800652 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:41.801354 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:41.801355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:41.801355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:41.801360 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:41.801377 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:41.801382 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:41.801442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:41.801580 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 00:16:41.801593 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:16:41.801939 21241 sys_catalog.cc:455] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c876998869644d32981729b2396a7954. Latest consensus state: current_term: 1 leader_uuid: "c876998869644d32981729b2396a7954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } } }
I20260620 00:16:41.802075 21241 sys_catalog.cc:458] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:41.802048 21243 sys_catalog.cc:455] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c876998869644d32981729b2396a7954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c876998869644d32981729b2396a7954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36773 } } }
I20260620 00:16:41.802292 21243 sys_catalog.cc:458] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:41.802512 21250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:41.803061 21250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:41.804759 21250 catalog_manager.cc:1347] Generated new cluster ID: 8e33115a4bab40df9b72a01205e422ae
I20260620 00:16:41.804826 21250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:41.808157 21250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:41.808339 21250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:41.818506 21250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c876998869644d32981729b2396a7954: Generated new TSK 0
I20260620 00:16:41.818787 21250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:41.822317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21163
2026-06-20T00:16:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:41 dist-test-slave-28bv krb5kdc[21262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:41 dist-test-slave-28bv krb5kdc[21262](info): set up 2 sockets
Jun 20 00:16:41 dist-test-slave-28bv krb5kdc[21262](info): commencing operation
krb5kdc: starting...
W20260620 00:16:42.904847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:42 dist-test-slave-28bv krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:43.044800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:43.136444 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:43.136835 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:43.136880 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:43.138329 21275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:43.138429 21275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:43.138453 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:43.138476 21275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:43.138505 21275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:43.140518 21275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:43.140925 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:43.141188 21275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:43.146692 21281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:43.146684 21280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:43.146692 21283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:43.146920 21275 server_base.cc:1034] running on GCE node
I20260620 00:16:43.147286 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:43.147644 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:43.148801 21275 hybrid_clock.cc:648] HybridClock initialized: now 1781914603148754 us; error 62 us; skew 500 ppm
Jun 20 00:16:43 dist-test-slave-28bv krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:43.151675 21275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:43.152216 21275 webserver.cc:460] Webserver started at http://127.0.0.1:45315/ using document root <none> and password file <none>
I20260620 00:16:43.152422 21275 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:43.152467 21275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:43.152630 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:43.153790 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "7b52eeb8d6b0457aae897a4367978139"
format_stamp: "Formatted at 2026-06-20 00:16:43 on dist-test-slave-28bv"
server_key: "0e4b652fea16e1c3e54e3a2f33fb2c06"
server_key_iv: "80b08458c02921a942f2979a73d3a2a1"
server_key_version: "encryptionkey@0"
I20260620 00:16:43.154131 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b52eeb8d6b0457aae897a4367978139"
format_stamp: "Formatted at 2026-06-20 00:16:43 on dist-test-slave-28bv"
server_key: "0e4b652fea16e1c3e54e3a2f33fb2c06"
server_key_iv: "80b08458c02921a942f2979a73d3a2a1"
server_key_version: "encryptionkey@0"
I20260620 00:16:43.155622 21275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:43.156631 21290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:43.156850 21275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:43.156976 21275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "7b52eeb8d6b0457aae897a4367978139"
format_stamp: "Formatted at 2026-06-20 00:16:43 on dist-test-slave-28bv"
server_key: "0e4b652fea16e1c3e54e3a2f33fb2c06"
server_key_iv: "80b08458c02921a942f2979a73d3a2a1"
server_key_version: "encryptionkey@0"
I20260620 00:16:43.157076 21275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:43.179013 21275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:43.181761 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:43.181967 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:43.186736 21275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39431
I20260620 00:16:43.186758 21342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39431 every 8 connection(s)
I20260620 00:16:43.187218 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:43.188508 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:43.191126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21275
I20260620 00:16:43.191289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:43.191488 392 external_mini_cluster.cc:1428] Setting key 24614f05c03ccbe9cf64100519d1062c
I20260620 00:16:43.191524 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: Bootstrap starting.
I20260620 00:16:43.192363 21343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:43.192730 21343 log.cc:824] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:43.193641 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: No bootstrap required, opened a new log
I20260620 00:16:43.196365 21343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260620 00:16:43.196717 21343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:43.196792 21343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b52eeb8d6b0457aae897a4367978139, State: Initialized, Role: FOLLOWER
I20260620 00:16:43.196985 21343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260620 00:16:43.197096 21343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:43.197149 21343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:43.197204 21343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:43.198210 21343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260620 00:16:43.198352 21343 leader_election.cc:302] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b52eeb8d6b0457aae897a4367978139; no voters:
I20260620 00:16:43.198571 21343 leader_election.cc:288] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:43.198678 21348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:43.198912 21348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [term 1 LEADER]: Becoming Leader. State: Replica: 7b52eeb8d6b0457aae897a4367978139, State: Running, Role: LEADER
I20260620 00:16:43.198925 21343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:43.199116 21348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } }
I20260620 00:16:43.199527 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b52eeb8d6b0457aae897a4367978139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } } }
I20260620 00:16:43.199623 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:43.199800 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b52eeb8d6b0457aae897a4367978139. Latest consensus state: current_term: 1 leader_uuid: "7b52eeb8d6b0457aae897a4367978139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b52eeb8d6b0457aae897a4367978139" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39431 } } }
I20260620 00:16:43.199862 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139 [sys.catalog]: This master's current role is: LEADER
Jun 20 00:16:43 dist-test-slave-28bv krb5kdc[21262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260620 00:16:43.201217 21364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:43.201287 21364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:43.201393 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:43.201606 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:43.203620 21354 catalog_manager.cc:1347] Generated new cluster ID: ff74a1ce2b684df19f2d3d7b9924a40b
I20260620 00:16:43.203706 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:43.205986 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:43.192124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:44051 (local address 127.0.0.1:39431)
0620 00:16:43.192341 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:43.192350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:43.192379 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:16:43.193084 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:43.193094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:43.193157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:43.193309 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:43.194250 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:43.195245 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:43.196611 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:43.197345 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:43.197455 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:43.201041 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:43.201046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:43.201047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:43.201061 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:43.203539 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:43.204185 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:43.204189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:43.204189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:43.204306 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:43.204596 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:43.204597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:43.204597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:43.204946 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:43.205108 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:43.205612 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 00:16:43.205730 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260620 00:16:43.207337 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:43.210253 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:43.210845 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:43.218585 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b52eeb8d6b0457aae897a4367978139: Generated new TSK 0
I20260620 00:16:43.218971 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:43.221824 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:43.209196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:36161 (local address 127.0.0.1:39431)
0620 00:16:43.209247 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:43.209249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:43.212720 (+ 3471us) server_negotiation.cc:407] Connection header received
0620 00:16:43.212801 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:43.212802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:43.212818 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:43.212903 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:43.213710 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:43.214740 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:43.216300 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:43.216718 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:43.216798 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:43.218558 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:43.218559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:43.218560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:43.218570 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:43.220301 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:43.220876 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:43.220878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:43.220879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:43.220946 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:43.221203 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:43.221205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:43.221205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:43.221335 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:43.221480 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:43.221630 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:16:43.221703 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:16:43.224643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21275
W20260620 00:16:43.229202 21260 connection.cc:535] client connection to 127.0.0.1:39431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39431): Connection reset by peer (error 104)
2026-06-20T00:16:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:43 dist-test-slave-28bv krb5kdc[21374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:43 dist-test-slave-28bv krb5kdc[21374](info): set up 2 sockets
Jun 20 00:16:43 dist-test-slave-28bv krb5kdc[21374](info): commencing operation
krb5kdc: starting...
W20260620 00:16:44.329771 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:44 dist-test-slave-28bv krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:44.470624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:44.559152 21387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:44.559427 21387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:44.559458 21387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:44.560812 21387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:44.560863 21387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:44.560879 21387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:44.560894 21387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:44.560906 21387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:44.562495 21387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:44.562726 21387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:44.562912 21387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:44.569075 21395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:44.569099 21392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:44.569088 21393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:44.569135 21387 server_base.cc:1034] running on GCE node
I20260620 00:16:44.569500 21387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:44.569908 21387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:44.571126 21387 hybrid_clock.cc:648] HybridClock initialized: now 1781914604571098 us; error 62 us; skew 500 ppm
Jun 20 00:16:44 dist-test-slave-28bv krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:44.574096 21387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:44.574600 21387 webserver.cc:460] Webserver started at http://127.0.0.1:36185/ using document root <none> and password file <none>
I20260620 00:16:44.574848 21387 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:44.574920 21387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:44.575075 21387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:44.576234 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3"
format_stamp: "Formatted at 2026-06-20 00:16:44 on dist-test-slave-28bv"
server_key: "56487264a6b76c4dc969e6e53213fcb4"
server_key_iv: "778e0f6ccdb6322eac9b4f664c0edff7"
server_key_version: "encryptionkey@0"
I20260620 00:16:44.576730 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3"
format_stamp: "Formatted at 2026-06-20 00:16:44 on dist-test-slave-28bv"
server_key: "56487264a6b76c4dc969e6e53213fcb4"
server_key_iv: "778e0f6ccdb6322eac9b4f664c0edff7"
server_key_version: "encryptionkey@0"
I20260620 00:16:44.578190 21387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:44.579207 21402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:44.579552 21387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:44.579761 21387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3"
format_stamp: "Formatted at 2026-06-20 00:16:44 on dist-test-slave-28bv"
server_key: "56487264a6b76c4dc969e6e53213fcb4"
server_key_iv: "778e0f6ccdb6322eac9b4f664c0edff7"
server_key_version: "encryptionkey@0"
I20260620 00:16:44.579871 21387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:44.594655 21387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:44.597667 21387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:44.597877 21387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:44.602490 21387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45805
I20260620 00:16:44.602530 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45805 every 8 connection(s)
I20260620 00:16:44.602908 21387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:44.603896 21455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:44.606808 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3: Bootstrap starting.
I20260620 00:16:44.607249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21387
I20260620 00:16:44.607491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:44.607575 21455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:44.607764 392 external_mini_cluster.cc:1428] Setting key 7c62584e8c9d4667e343cccf1839d69e
I20260620 00:16:44.607926 21455 log.cc:824] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:44.609007 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3: No bootstrap required, opened a new log
I20260620 00:16:44.611970 21455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } }
I20260620 00:16:44.612169 21455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:44.612196 21455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416f31eb4ffd4d1d80ef4ba249c0caa3, State: Initialized, Role: FOLLOWER
I20260620 00:16:44.612349 21455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } }
I20260620 00:16:44.612442 21455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:44.612490 21455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:44.612572 21455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:44.613467 21455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } }
I20260620 00:16:44.613610 21455 leader_election.cc:302] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 416f31eb4ffd4d1d80ef4ba249c0caa3; no voters:
I20260620 00:16:44.613823 21455 leader_election.cc:288] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:44.613993 21460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:44.614343 21460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [term 1 LEADER]: Becoming Leader. State: Replica: 416f31eb4ffd4d1d80ef4ba249c0caa3, State: Running, Role: LEADER
I20260620 00:16:44.614538 21455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:44.614545 21460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } }
I20260620 00:16:44.615060 21462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416f31eb4ffd4d1d80ef4ba249c0caa3. Latest consensus state: current_term: 1 leader_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } } }
I20260620 00:16:44.615242 21462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:44.615217 21461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "416f31eb4ffd4d1d80ef4ba249c0caa3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45805 } } }
I20260620 00:16:44.615379 21461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:44.615695 21469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:44.615767 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:44.608427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:41643 (local address 127.0.0.1:45805)
0620 00:16:44.608652 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:44.608663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:44.608701 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:16:44.609385 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:44.609395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:44.609469 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:44.609655 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:44.609918 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:44.611507 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:44.612751 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:44.613417 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:44.613560 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:44.614808 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:44.614813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:44.614813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:44.615061 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:44.615108 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:44.615126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:44.615221 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:44.615452 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 00:16:44.615489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260620 00:16:44.616307 21469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:44.616882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:44.621110 21469 catalog_manager.cc:1347] Generated new cluster ID: fd242c193d09471e8baa2f0c85551c84
I20260620 00:16:44.621207 21469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:44.624333 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:44.618682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:35193 (local address 127.0.0.1:45805)
0620 00:16:44.618722 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:44.618724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:44.621251 (+ 2527us) server_negotiation.cc:407] Connection header received
0620 00:16:44.621334 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:44.621336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:44.621348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:44.621402 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:44.621614 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:44.622173 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:44.622955 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:44.623186 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:44.623273 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:44.624032 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:44.624033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:44.624033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:44.624038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:44.624055 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:44.624061 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:44.624135 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:44.624220 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 00:16:44.624231 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:16:44.634789 21469 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:44.635604 21469 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:44.643035 21469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416f31eb4ffd4d1d80ef4ba249c0caa3: Generated new TSK 0
I20260620 00:16:44.643299 21469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:44.653607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21387
W20260620 00:16:44.658509 21372 connection.cc:535] client connection to 127.0.0.1:45805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45805): Connection reset by peer (error 104)
2026-06-20T00:16:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:44 dist-test-slave-28bv krb5kdc[21486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:44 dist-test-slave-28bv krb5kdc[21486](info): set up 2 sockets
Jun 20 00:16:44 dist-test-slave-28bv krb5kdc[21486](info): commencing operation
krb5kdc: starting...
W20260620 00:16:45.720803 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:45 dist-test-slave-28bv krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:45.849767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:45.931784 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:45.932015 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:45.932039 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:45.933423 21499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:45.933480 21499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:45.933495 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:45.933508 21499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:45.933521 21499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:45.935091 21499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:45.935371 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:45.935631 21499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:45.940779 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:45.940765 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:45.940786 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:45.940765 21499 server_base.cc:1034] running on GCE node
I20260620 00:16:45.941373 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:45.941676 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:45.942830 21499 hybrid_clock.cc:648] HybridClock initialized: now 1781914605942808 us; error 38 us; skew 500 ppm
Jun 20 00:16:45 dist-test-slave-28bv krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:45.949422 21499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:45.949971 21499 webserver.cc:460] Webserver started at http://127.0.0.1:41457/ using document root <none> and password file <none>
I20260620 00:16:45.950263 21499 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:45.950335 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:45.950484 21499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:45.951563 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "cb004402268d4cc189d302ee9bf99e80"
format_stamp: "Formatted at 2026-06-20 00:16:45 on dist-test-slave-28bv"
server_key: "d51e2bb3fbb23d8f67e981b13662f343"
server_key_iv: "2410432c3b783c7fcd994bca41c210a9"
server_key_version: "encryptionkey@0"
I20260620 00:16:45.951932 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb004402268d4cc189d302ee9bf99e80"
format_stamp: "Formatted at 2026-06-20 00:16:45 on dist-test-slave-28bv"
server_key: "d51e2bb3fbb23d8f67e981b13662f343"
server_key_iv: "2410432c3b783c7fcd994bca41c210a9"
server_key_version: "encryptionkey@0"
I20260620 00:16:45.953384 21499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:45.954197 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:45.954397 21499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:16:45.954507 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "cb004402268d4cc189d302ee9bf99e80"
format_stamp: "Formatted at 2026-06-20 00:16:45 on dist-test-slave-28bv"
server_key: "d51e2bb3fbb23d8f67e981b13662f343"
server_key_iv: "2410432c3b783c7fcd994bca41c210a9"
server_key_version: "encryptionkey@0"
I20260620 00:16:45.954582 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:45.958060 21499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:45.960378 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:45.960618 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:45.964730 21499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40725
I20260620 00:16:45.964763 21566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40725 every 8 connection(s)
I20260620 00:16:45.965108 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:45.966076 21567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:45.968318 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80: Bootstrap starting.
I20260620 00:16:45.968935 21567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:45.969168 21567 log.cc:824] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:45.969823 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80: No bootstrap required, opened a new log
I20260620 00:16:45.971356 21567 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } }
I20260620 00:16:45.971493 21567 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:45.971516 21567 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb004402268d4cc189d302ee9bf99e80, State: Initialized, Role: FOLLOWER
I20260620 00:16:45.971669 21567 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } }
I20260620 00:16:45.971751 21567 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:45.971773 21567 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:45.971801 21567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:45.972285 21567 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } }
I20260620 00:16:45.972365 21567 leader_election.cc:302] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb004402268d4cc189d302ee9bf99e80; no voters:
I20260620 00:16:45.972482 21567 leader_election.cc:288] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:45.972584 21570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:45.972756 21567 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:45.972815 21570 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [term 1 LEADER]: Becoming Leader. State: Replica: cb004402268d4cc189d302ee9bf99e80, State: Running, Role: LEADER
I20260620 00:16:45.973006 21570 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } }
I20260620 00:16:45.973588 21572 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb004402268d4cc189d302ee9bf99e80. Latest consensus state: current_term: 1 leader_uuid: "cb004402268d4cc189d302ee9bf99e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } } }
I20260620 00:16:45.973673 21572 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:45.973771 21571 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb004402268d4cc189d302ee9bf99e80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb004402268d4cc189d302ee9bf99e80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40725 } } }
I20260620 00:16:45.973867 21571 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:45.973863 21579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:45.974445 21579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:45.974617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21499
I20260620 00:16:45.974731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:45.974924 392 external_mini_cluster.cc:1428] Setting key ff340199d19817a54dc3ab9b1c48d969
I20260620 00:16:45.976011 21579 catalog_manager.cc:1347] Generated new cluster ID: abe42703d922400bab69d5b6243f8fca
I20260620 00:16:45.976076 21579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:45.978266 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:45.975396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:38647 (local address 127.0.0.1:40725)
0620 00:16:45.975553 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:45.975563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:45.976807 (+ 1244us) server_negotiation.cc:407] Connection header received
0620 00:16:45.976968 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:45.976974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:45.976998 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:45.977145 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:45.977824 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:45.977828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:45.977828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:45.977832 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:45.977878 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:45.977889 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:45.977963 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:45.978030 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:16:45.978079 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 00:16:45.988142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:45.991142 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:45.989848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:54037 (local address 127.0.0.1:40725)
0620 00:16:45.989897 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:45.989899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:45.990064 (+ 165us) server_negotiation.cc:407] Connection header received
0620 00:16:45.990139 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:45.990141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:45.990148 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:45.990206 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:45.990887 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:45.990889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:45.990889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:45.990894 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:45.990911 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:45.990916 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:45.990970 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:45.991021 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 00:16:45.991038 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:16:46.001734 21579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:46.001955 21579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:46.013274 21579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb004402268d4cc189d302ee9bf99e80: Generated new TSK 0
I20260620 00:16:46.013499 21579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:46.024381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21499
2026-06-20T00:16:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:46 dist-test-slave-28bv krb5kdc[21598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:46 dist-test-slave-28bv krb5kdc[21598](info): set up 2 sockets
Jun 20 00:16:46 dist-test-slave-28bv krb5kdc[21598](info): commencing operation
krb5kdc: starting...
W20260620 00:16:47.072798 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:47.196177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:47.277449 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:47.277679 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:47.277704 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:47.278935 21611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:47.278985 21611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:47.279000 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:47.279012 21611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:47.279024 21611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:47.280609 21611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:47.280893 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:47.281147 21611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:47.286476 21616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:47.286535 21611 server_base.cc:1034] running on GCE node
W20260620 00:16:47.286494 21617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:47.286494 21619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:47.287010 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:47.287299 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:47.288455 21611 hybrid_clock.cc:648] HybridClock initialized: now 1781914607288432 us; error 44 us; skew 500 ppm
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:47.291724 21611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:47.292249 21611 webserver.cc:460] Webserver started at http://127.0.0.1:38627/ using document root <none> and password file <none>
I20260620 00:16:47.292498 21611 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:47.292594 21611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:47.292769 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:47.293859 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "11be1a30f13245a191254ed47619247f"
format_stamp: "Formatted at 2026-06-20 00:16:47 on dist-test-slave-28bv"
server_key: "0778de34afc8ae73928f75acd0c2bbfd"
server_key_iv: "252f56b1b71791285f4daec52661a3df"
server_key_version: "encryptionkey@0"
I20260620 00:16:47.294225 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "11be1a30f13245a191254ed47619247f"
format_stamp: "Formatted at 2026-06-20 00:16:47 on dist-test-slave-28bv"
server_key: "0778de34afc8ae73928f75acd0c2bbfd"
server_key_iv: "252f56b1b71791285f4daec52661a3df"
server_key_version: "encryptionkey@0"
I20260620 00:16:47.295611 21611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:47.296384 21626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:47.296648 21611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:47.296766 21611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "11be1a30f13245a191254ed47619247f"
format_stamp: "Formatted at 2026-06-20 00:16:47 on dist-test-slave-28bv"
server_key: "0778de34afc8ae73928f75acd0c2bbfd"
server_key_iv: "252f56b1b71791285f4daec52661a3df"
server_key_version: "encryptionkey@0"
I20260620 00:16:47.296847 21611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:47.321593 21611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:47.323936 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:47.324108 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:47.328444 21611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42941
I20260620 00:16:47.328482 21678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42941 every 8 connection(s)
I20260620 00:16:47.328923 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:47.329761 21679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:47.331265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21611
I20260620 00:16:47.331406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:47.331624 392 external_mini_cluster.cc:1428] Setting key 2d52f41e85e28459b8a55f86fae891d7
I20260620 00:16:47.332311 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f: Bootstrap starting.
I20260620 00:16:47.333060 21679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:47.333375 21679 log.cc:824] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:47.334100 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f: No bootstrap required, opened a new log
I20260620 00:16:47.336354 21679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } }
I20260620 00:16:47.336526 21679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:47.336601 21679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11be1a30f13245a191254ed47619247f, State: Initialized, Role: FOLLOWER
I20260620 00:16:47.336728 21679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } }
I20260620 00:16:47.336807 21679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:47.336839 21679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:47.336884 21679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:47.337483 21679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } }
I20260620 00:16:47.337615 21679 leader_election.cc:302] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11be1a30f13245a191254ed47619247f; no voters:
I20260620 00:16:47.337796 21679 leader_election.cc:288] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:47.337838 21684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:47.338006 21684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [term 1 LEADER]: Becoming Leader. State: Replica: 11be1a30f13245a191254ed47619247f, State: Running, Role: LEADER
I20260620 00:16:47.338076 21679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:47.338107 21684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } }
I20260620 00:16:47.338521 21686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11be1a30f13245a191254ed47619247f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } } }
I20260620 00:16:47.338534 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11be1a30f13245a191254ed47619247f. Latest consensus state: current_term: 1 leader_uuid: "11be1a30f13245a191254ed47619247f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11be1a30f13245a191254ed47619247f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42941 } } }
I20260620 00:16:47.338625 21686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:47.338629 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f [sys.catalog]: This master's current role is: LEADER
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:16:47.339613 21693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:47.340199 21693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:47.341846 21693 catalog_manager.cc:1347] Generated new cluster ID: 3192b66df2674f04b7a4934dde595fcb
I20260620 00:16:47.341900 21693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:47.343494 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:47.332214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:40501 (local address 127.0.0.1:42941)
0620 00:16:47.332431 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:47.332440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:47.332472 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:16:47.333126 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:47.333134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:47.333209 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:47.333365 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:47.334043 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:47.335413 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:47.336206 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:47.336837 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:47.336956 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:47.340063 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:47.340066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:47.340067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:47.340076 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:47.342059 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:47.342464 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:47.342466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:47.342467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:47.342542 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:47.342741 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:47.342741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:47.342741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:47.342944 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:47.343038 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:47.343179 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:16:47.343252 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260620 00:16:47.344399 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:47.352825 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:47.345800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:47713 (local address 127.0.0.1:42941)
0620 00:16:47.345840 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:47.345842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:47.345938 (+ 96us) server_negotiation.cc:407] Connection header received
0620 00:16:47.345986 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:47.345987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:47.345995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:47.346040 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:47.346629 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:47.347130 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:47.348089 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:47.348312 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:47.348370 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:47.349750 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:47.349752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:47.349752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:47.349759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:47.351370 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:47.351805 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:47.351806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:47.351806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:47.351869 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:47.352138 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:47.352139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:47.352139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:47.352273 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:47.352353 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:47.352655 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 00:16:47.352727 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:47.362112 21693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:47.362689 21693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:47.366334 21693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11be1a30f13245a191254ed47619247f: Generated new TSK 0
I20260620 00:16:47.366568 21693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:47.371013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21611
W20260620 00:16:47.375639 21596 connection.cc:535] client connection to 127.0.0.1:42941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42941): Connection reset by peer (error 104)
2026-06-20T00:16:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21710](info): set up 2 sockets
Jun 20 00:16:47 dist-test-slave-28bv krb5kdc[21710](info): commencing operation
krb5kdc: starting...
W20260620 00:16:48.424876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:48 dist-test-slave-28bv krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:48.555904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:48.638334 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:48.638608 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:48.638649 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:48.639890 21723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:48.639967 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:48.639994 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:48.640026 21723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:48.640049 21723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:48.641767 21723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:48.642067 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:48.642309 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:48.647302 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:48.647321 21731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:48.647382 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:48.647567 21723 server_base.cc:1034] running on GCE node
I20260620 00:16:48.647886 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:48.648211 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:48.649367 21723 hybrid_clock.cc:648] HybridClock initialized: now 1781914608649363 us; error 41 us; skew 500 ppm
Jun 20 00:16:48 dist-test-slave-28bv krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:48.652002 21723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:48.652410 21723 webserver.cc:460] Webserver started at http://127.0.0.1:33311/ using document root <none> and password file <none>
I20260620 00:16:48.652690 21723 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:48.652740 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:48.652833 21723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:48.653852 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "6df2d46dba3b41bea562f8d812b2b810"
format_stamp: "Formatted at 2026-06-20 00:16:48 on dist-test-slave-28bv"
server_key: "3546eaf18d437bccf99609d64f7052aa"
server_key_iv: "5d9f193570e760a45fefaa67ba494911"
server_key_version: "encryptionkey@0"
I20260620 00:16:48.654171 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "6df2d46dba3b41bea562f8d812b2b810"
format_stamp: "Formatted at 2026-06-20 00:16:48 on dist-test-slave-28bv"
server_key: "3546eaf18d437bccf99609d64f7052aa"
server_key_iv: "5d9f193570e760a45fefaa67ba494911"
server_key_version: "encryptionkey@0"
I20260620 00:16:48.655480 21723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:48.656240 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:48.656420 21723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:48.656572 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "6df2d46dba3b41bea562f8d812b2b810"
format_stamp: "Formatted at 2026-06-20 00:16:48 on dist-test-slave-28bv"
server_key: "3546eaf18d437bccf99609d64f7052aa"
server_key_iv: "5d9f193570e760a45fefaa67ba494911"
server_key_version: "encryptionkey@0"
I20260620 00:16:48.656669 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:48.670135 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:48.672312 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:48.672490 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:48.676577 21723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46601
I20260620 00:16:48.676597 21790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46601 every 8 connection(s)
I20260620 00:16:48.677004 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:48.678007 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:48.680276 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810: Bootstrap starting.
I20260620 00:16:48.680838 21791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:48.681047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21723
I20260620 00:16:48.681175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:48.681176 21791 log.cc:824] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:48.681378 392 external_mini_cluster.cc:1428] Setting key 1f6cc0dba76951e6d3bc23fc655a7880
I20260620 00:16:48.681975 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810: No bootstrap required, opened a new log
I20260620 00:16:48.683631 21791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260620 00:16:48.683800 21791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:48.683847 21791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df2d46dba3b41bea562f8d812b2b810, State: Initialized, Role: FOLLOWER
I20260620 00:16:48.683969 21791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260620 00:16:48.684048 21791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:48.684085 21791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:48.684140 21791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:48.684778 21791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260620 00:16:48.684866 21791 leader_election.cc:302] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df2d46dba3b41bea562f8d812b2b810; no voters:
I20260620 00:16:48.685079 21791 leader_election.cc:288] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:48.685175 21796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:48.685340 21796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [term 1 LEADER]: Becoming Leader. State: Replica: 6df2d46dba3b41bea562f8d812b2b810, State: Running, Role: LEADER
I20260620 00:16:48.685397 21791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:48.685422 21796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } }
I20260620 00:16:48.685746 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df2d46dba3b41bea562f8d812b2b810. Latest consensus state: current_term: 1 leader_uuid: "6df2d46dba3b41bea562f8d812b2b810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } } }
I20260620 00:16:48.685830 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:48.686049 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df2d46dba3b41bea562f8d812b2b810" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df2d46dba3b41bea562f8d812b2b810" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46601 } } }
I20260620 00:16:48.686134 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:48.687093 21801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:48.687569 21801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:48.689235 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:48.681895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:47737 (local address 127.0.0.1:46601)
0620 00:16:48.682118 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:48.682125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:48.682152 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:16:48.683624 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:48.683632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:48.683690 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:48.683866 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:48.684109 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:48.685727 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:48.686991 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:48.687596 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:48.687708 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:48.688595 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:48.688599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:48.688599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:48.688786 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:48.688824 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:48.688835 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:48.688937 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:48.689021 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 00:16:48.689052 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
I20260620 00:16:48.689337 21801 catalog_manager.cc:1347] Generated new cluster ID: 66536bfd2d474c3b841c9c1211e82777
I20260620 00:16:48.689393 21801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:48.693645 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:48.698851 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:48.695243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:56139 (local address 127.0.0.1:46601)
0620 00:16:48.695281 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:48.695283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:48.695315 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:16:48.695394 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:48.695394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:48.695405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:48.695463 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:48.695593 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:48.696370 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:48.697393 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:48.697724 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:48.697802 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:48.698549 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:48.698551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:48.698551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:48.698556 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:48.698574 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:48.698578 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:48.698644 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:48.698730 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:16:48.698745 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:16:48.699625 21801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:48.700098 21801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:48.704022 21801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df2d46dba3b41bea562f8d812b2b810: Generated new TSK 0
I20260620 00:16:48.704203 21801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:48.706496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21723
W20260620 00:16:48.710376 21708 connection.cc:535] client connection to 127.0.0.1:46601 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46601): Connection reset by peer (error 104)
2026-06-20T00:16:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:48 dist-test-slave-28bv krb5kdc[21822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:48 dist-test-slave-28bv krb5kdc[21822](info): set up 2 sockets
Jun 20 00:16:48 dist-test-slave-28bv krb5kdc[21822](info): commencing operation
krb5kdc: starting...
W20260620 00:16:49.760834 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:49 dist-test-slave-28bv krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:16:49.888404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:49.969832 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:49.970096 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:49.970120 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:49.971251 21835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:49.971294 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:49.971309 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:49.971323 21835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:49.971334 21835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:49.972838 21835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42459
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:49.973140 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:49.973368 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:49.978680 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:49.978683 21843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:49.978683 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:49.979012 21835 server_base.cc:1034] running on GCE node
I20260620 00:16:49.979352 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:49.979672 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:49.980854 21835 hybrid_clock.cc:648] HybridClock initialized: now 1781914609980831 us; error 29 us; skew 500 ppm
Jun 20 00:16:49 dist-test-slave-28bv krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:49.983551 21835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:16:49.984041 21835 webserver.cc:460] Webserver started at http://127.0.0.1:34345/ using document root <none> and password file <none>
I20260620 00:16:49.984237 21835 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:49.984277 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:49.984369 21835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:49.985494 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "a4e829f02404413985a96649417b4131"
format_stamp: "Formatted at 2026-06-20 00:16:49 on dist-test-slave-28bv"
server_key: "8b4e00b9fe92f10f632b342c238016bf"
server_key_iv: "6e34a8892e6da88582b150691ef2941a"
server_key_version: "encryptionkey@0"
I20260620 00:16:49.985808 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4e829f02404413985a96649417b4131"
format_stamp: "Formatted at 2026-06-20 00:16:49 on dist-test-slave-28bv"
server_key: "8b4e00b9fe92f10f632b342c238016bf"
server_key_iv: "6e34a8892e6da88582b150691ef2941a"
server_key_version: "encryptionkey@0"
I20260620 00:16:49.987097 21835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:16:49.987820 21850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:49.987993 21835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:49.988082 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "a4e829f02404413985a96649417b4131"
format_stamp: "Formatted at 2026-06-20 00:16:49 on dist-test-slave-28bv"
server_key: "8b4e00b9fe92f10f632b342c238016bf"
server_key_iv: "6e34a8892e6da88582b150691ef2941a"
server_key_version: "encryptionkey@0"
I20260620 00:16:49.988139 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:49.991485 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:49.993808 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:49.993918 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:49.997720 21835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42459
I20260620 00:16:49.997776 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42459 every 8 connection(s)
I20260620 00:16:49.998152 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:49.999194 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:50.002139 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131: Bootstrap starting.
I20260620 00:16:50.002808 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:50.003110 21903 log.cc:824] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:50.003428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21835
I20260620 00:16:50.003573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:50.003875 392 external_mini_cluster.cc:1428] Setting key a1642a93d4b8db2549011e0609aa3c95
I20260620 00:16:50.003924 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131: No bootstrap required, opened a new log
I20260620 00:16:50.006390 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 00:16:50.006603 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:50.006650 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e829f02404413985a96649417b4131, State: Initialized, Role: FOLLOWER
I20260620 00:16:50.006834 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 00:16:50.006939 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:50.006966 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:50.007047 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:50.008219 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:50.004503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:40791 (local address 127.0.0.1:42459)
0620 00:16:50.004757 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:50.004765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:50.004791 (+ 26us) server_negotiation.cc:407] Connection header received
0620 00:16:50.006375 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:50.006384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:50.006419 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:50.006568 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:50.007673 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:50.007677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:50.007677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:50.007681 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:50.007721 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:50.007733 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:50.007790 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:50.007961 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:16:50.008020 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":154,"threads_started":1}
I20260620 00:16:50.008752 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 00:16:50.008929 21903 leader_election.cc:302] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4e829f02404413985a96649417b4131; no voters:
I20260620 00:16:50.009157 21903 leader_election.cc:288] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:50.009181 21908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:50.009375 21908 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [term 1 LEADER]: Becoming Leader. State: Replica: a4e829f02404413985a96649417b4131, State: Running, Role: LEADER
I20260620 00:16:50.009516 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:50.009542 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:50.009516 21908 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } }
I20260620 00:16:50.009991 21908 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4e829f02404413985a96649417b4131. Latest consensus state: current_term: 1 leader_uuid: "a4e829f02404413985a96649417b4131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260620 00:16:50.010058 21908 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:50.010342 21912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:50.010866 21909 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4e829f02404413985a96649417b4131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e829f02404413985a96649417b4131" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42459 } } }
I20260620 00:16:50.010946 21912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:50.011006 21909 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:50.013793 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:50.011153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.160:45569 (local address 127.0.0.1:42459)
0620 00:16:50.011190 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:50.011192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:50.012739 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 00:16:50.012809 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:50.012809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:50.012814 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:50.012915 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:50.013578 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:50.013580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:50.013580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:50.013585 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:50.013601 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:50.013606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:50.013653 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:50.013706 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:16:50.013723 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:16:50.020164 21912 catalog_manager.cc:1347] Generated new cluster ID: 82f5cead72c948e7b5d651dfa3177006
I20260620 00:16:50.020253 21912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:50.029604 21912 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:50.029824 21912 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:50.032791 21912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4e829f02404413985a96649417b4131: Generated new TSK 0
I20260620 00:16:50.033013 21912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:50.039961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21835
2026-06-20T00:16:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:50 dist-test-slave-28bv krb5kdc[21934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:50 dist-test-slave-28bv krb5kdc[21934](info): set up 2 sockets
Jun 20 00:16:50 dist-test-slave-28bv krb5kdc[21934](info): commencing operation
krb5kdc: starting...
W20260620 00:16:51.093230 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:51.232049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38373
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:51.314769 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:51.315044 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:51.315085 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:51.316300 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:51.316376 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:51.316402 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:51.316430 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:51.316461 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:51.318005 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:51.318298 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:51.318547 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:51.323699 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:51.323681 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:51.323732 21947 server_base.cc:1034] running on GCE node
W20260620 00:16:51.323717 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:51.324146 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:51.324456 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:51.325660 21947 hybrid_clock.cc:648] HybridClock initialized: now 1781914611325642 us; error 44 us; skew 500 ppm
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:51.333108 21947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:51.333689 21947 webserver.cc:460] Webserver started at http://127.0.98.62:33681/ using document root <none> and password file <none>
I20260620 00:16:51.333949 21947 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:51.334000 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:51.334144 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:51.335426 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "9296e3fc18984a5195dad8b2fde77249"
format_stamp: "Formatted at 2026-06-20 00:16:51 on dist-test-slave-28bv"
server_key: "02978ae913ba728bfb098c026b56bf72"
server_key_iv: "c3a31a579617e65551cd29dca23a181e"
server_key_version: "encryptionkey@0"
I20260620 00:16:51.335819 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "9296e3fc18984a5195dad8b2fde77249"
format_stamp: "Formatted at 2026-06-20 00:16:51 on dist-test-slave-28bv"
server_key: "02978ae913ba728bfb098c026b56bf72"
server_key_iv: "c3a31a579617e65551cd29dca23a181e"
server_key_version: "encryptionkey@0"
I20260620 00:16:51.337610 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 00:16:51.338639 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:51.338936 21947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:51.339063 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "9296e3fc18984a5195dad8b2fde77249"
format_stamp: "Formatted at 2026-06-20 00:16:51 on dist-test-slave-28bv"
server_key: "02978ae913ba728bfb098c026b56bf72"
server_key_iv: "c3a31a579617e65551cd29dca23a181e"
server_key_version: "encryptionkey@0"
I20260620 00:16:51.339164 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:51.351706 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:51.354231 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:51.354430 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:51.359336 21947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38373
I20260620 00:16:51.359452 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38373 every 8 connection(s)
I20260620 00:16:51.359864 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:51.360870 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:51.363288 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: Bootstrap starting.
I20260620 00:16:51.363932 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:51.364234 22015 log.cc:824] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:51.365123 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: No bootstrap required, opened a new log
I20260620 00:16:51.366866 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } }
I20260620 00:16:51.367062 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:51.367113 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9296e3fc18984a5195dad8b2fde77249, State: Initialized, Role: FOLLOWER
I20260620 00:16:51.367290 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } }
I20260620 00:16:51.367395 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:51.367441 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:51.367491 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:51.367615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 21947
I20260620 00:16:51.367741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:51.367939 392 external_mini_cluster.cc:1428] Setting key 28bda0c3399058a1d123a628417c9558
I20260620 00:16:51.368302 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } }
I20260620 00:16:51.368499 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9296e3fc18984a5195dad8b2fde77249; no voters:
I20260620 00:16:51.368696 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:51.368728 22020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:51.368834 22020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [term 1 LEADER]: Becoming Leader. State: Replica: 9296e3fc18984a5195dad8b2fde77249, State: Running, Role: LEADER
I20260620 00:16:51.368984 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:51.369000 22020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } }
I20260620 00:16:51.369647 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9296e3fc18984a5195dad8b2fde77249. Latest consensus state: current_term: 1 leader_uuid: "9296e3fc18984a5195dad8b2fde77249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } } }
I20260620 00:16:51.370151 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:51.370060 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9296e3fc18984a5195dad8b2fde77249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9296e3fc18984a5195dad8b2fde77249" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38373 } } }
I20260620 00:16:51.370332 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249 [sys.catalog]: This master's current role is: LEADER
W20260620 00:16:51.371256 22036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:51.371331 22036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:51.371390 22030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:51.371557 22030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:51.373449 22030 catalog_manager.cc:1347] Generated new cluster ID: a82b63d186794c2eb3a0284bd229cd73
I20260620 00:16:51.373575 22030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[21934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:51.381614 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:51.368449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34465 (local address 127.0.98.62:38373)
0620 00:16:51.368697 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:51.368707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:51.368739 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:16:51.368872 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:51.368880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:51.368934 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:51.370004 (+ 1070us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:51.370120 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:51.371421 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:51.372732 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:51.373379 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:51.373524 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:51.376533 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:51.376565 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:51.376565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:51.376579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:51.379384 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:51.379937 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:51.379941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:51.379941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:51.380074 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:51.380413 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:51.380415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:51.380415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:51.380834 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:51.381025 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:51.381181 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:16:51.381310 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":131,"threads_started":1}
I20260620 00:16:51.382894 22030 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:51.382933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:51.383523 22030 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:51.393540 22030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9296e3fc18984a5195dad8b2fde77249: Generated new TSK 0
I20260620 00:16:51.393762 22030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:51.396699 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:51.384734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46229 (local address 127.0.98.62:38373)
0620 00:16:51.384784 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:51.384787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:51.388800 (+ 4013us) server_negotiation.cc:407] Connection header received
0620 00:16:51.388875 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:51.388877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:51.388890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:51.388958 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:51.389698 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:51.390795 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:51.391893 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:51.392209 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:51.392269 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:51.393892 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:51.393894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:51.393894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:51.393901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:51.395374 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:51.395818 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:51.395819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:51.395819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:51.395868 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:51.396136 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:51.396137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:51.396137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:51.396238 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:51.396317 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:51.396471 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:16:51.396522 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:51.398834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 21947
W20260620 00:16:51.403294 21932 connection.cc:535] client connection to 127.0.98.62:38373 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38373): Connection reset by peer (error 104)
2026-06-20T00:16:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[22046](info): set up 2 sockets
Jun 20 00:16:51 dist-test-slave-28bv krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260620 00:16:52.464661 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:52 dist-test-slave-28bv krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:52.595468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:52.677462 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:52.677701 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:52.677727 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:52.678980 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:52.679036 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:52.679050 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:52.679064 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:52.679075 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:52.680753 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:52.681023 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:52.681278 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:52.686877 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:52.686928 22059 server_base.cc:1034] running on GCE node
W20260620 00:16:52.686874 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:52.686903 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:52.687551 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:52.687867 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:52.689043 22059 hybrid_clock.cc:648] HybridClock initialized: now 1781914612689022 us; error 39 us; skew 500 ppm
Jun 20 00:16:52 dist-test-slave-28bv krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:52.691679 22059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:52.692211 22059 webserver.cc:460] Webserver started at http://127.0.98.62:37223/ using document root <none> and password file <none>
I20260620 00:16:52.692468 22059 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:52.692543 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:52.692724 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:52.693787 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "cf36855d67f04a4db8caaa2a93f557e9"
format_stamp: "Formatted at 2026-06-20 00:16:52 on dist-test-slave-28bv"
server_key: "fee8523e2dd352dafd02e6513777ed2f"
server_key_iv: "1d8b629aa23ff66341bd74230e79b588"
server_key_version: "encryptionkey@0"
I20260620 00:16:52.694310 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf36855d67f04a4db8caaa2a93f557e9"
format_stamp: "Formatted at 2026-06-20 00:16:52 on dist-test-slave-28bv"
server_key: "fee8523e2dd352dafd02e6513777ed2f"
server_key_iv: "1d8b629aa23ff66341bd74230e79b588"
server_key_version: "encryptionkey@0"
I20260620 00:16:52.695793 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:16:52.696841 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:52.697142 22059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:52.697279 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "cf36855d67f04a4db8caaa2a93f557e9"
format_stamp: "Formatted at 2026-06-20 00:16:52 on dist-test-slave-28bv"
server_key: "fee8523e2dd352dafd02e6513777ed2f"
server_key_iv: "1d8b629aa23ff66341bd74230e79b588"
server_key_version: "encryptionkey@0"
I20260620 00:16:52.697367 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:52.713441 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:52.715675 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:52.715863 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:52.720103 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43835
I20260620 00:16:52.720137 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43835 every 8 connection(s)
I20260620 00:16:52.720504 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:52.720778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22059
I20260620 00:16:52.720916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:52.721175 392 external_mini_cluster.cc:1428] Setting key d4c2781407f978f0d728cc7b1d5dc705
I20260620 00:16:52.721621 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:52.724406 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9: Bootstrap starting.
I20260620 00:16:52.725174 22127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:52.725492 22127 log.cc:824] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:52.726291 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9: No bootstrap required, opened a new log
I20260620 00:16:52.726986 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:52.721753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45059 (local address 127.0.98.62:43835)
0620 00:16:52.721962 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:52.721973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:52.722003 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:16:52.722424 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:52.722437 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:52.722483 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:52.722630 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:52.722870 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:52.723764 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:52.724637 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:52.725277 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:52.725410 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:52.726233 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:52.726237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:52.726237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:52.726466 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:52.726504 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:52.726519 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:52.726602 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:52.726725 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 00:16:52.726759 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
I20260620 00:16:52.728425 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:52.728734 22127 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260620 00:16:52.728902 22127 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:52.728955 22127 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf36855d67f04a4db8caaa2a93f557e9, State: Initialized, Role: FOLLOWER
I20260620 00:16:52.729089 22127 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260620 00:16:52.729187 22127 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:52.729233 22127 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:52.729281 22127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:52.729899 22127 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260620 00:16:52.730031 22127 leader_election.cc:302] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf36855d67f04a4db8caaa2a93f557e9; no voters:
I20260620 00:16:52.730229 22127 leader_election.cc:288] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:52.730283 22137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:52.730435 22137 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [term 1 LEADER]: Becoming Leader. State: Replica: cf36855d67f04a4db8caaa2a93f557e9, State: Running, Role: LEADER
I20260620 00:16:52.730530 22127 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:52.730571 22137 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } }
I20260620 00:16:52.730981 22137 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf36855d67f04a4db8caaa2a93f557e9. Latest consensus state: current_term: 1 leader_uuid: "cf36855d67f04a4db8caaa2a93f557e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260620 00:16:52.730989 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf36855d67f04a4db8caaa2a93f557e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf36855d67f04a4db8caaa2a93f557e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43835 } } }
I20260620 00:16:52.731134 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:52.731326 22137 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:52.731501 22146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:52.732064 22146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:52.733825 22146 catalog_manager.cc:1347] Generated new cluster ID: 66f0aa434c634108995a7e4efec39909
I20260620 00:16:52.733886 22146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:52.734077 22130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:52.730035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34055 (local address 127.0.98.62:43835)
0620 00:16:52.730082 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:52.730084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:52.730185 (+ 101us) server_negotiation.cc:407] Connection header received
0620 00:16:52.730262 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:52.730263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:52.730276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:52.730339 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:52.730558 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:52.731308 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:52.732464 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:52.732731 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:52.732806 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:52.733632 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:52.733634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:52.733634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:52.733639 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:52.733653 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:52.733658 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:52.733745 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:52.733998 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 00:16:52.734009 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:52.738199 22146 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:52.738708 22146 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:52.740476 22146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf36855d67f04a4db8caaa2a93f557e9: Generated new TSK 0
I20260620 00:16:52.740669 22146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:52.745390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22059
W20260620 00:16:52.750492 22044 connection.cc:535] client connection to 127.0.98.62:43835 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43835): Connection reset by peer (error 104)
2026-06-20T00:16:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:52 dist-test-slave-28bv krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:52 dist-test-slave-28bv krb5kdc[22158](info): set up 2 sockets
Jun 20 00:16:52 dist-test-slave-28bv krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260620 00:16:53.805004 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:53 dist-test-slave-28bv krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:53.931779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:54.019013 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:54.019289 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:54.019328 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:54.020532 22171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:54.020666 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:54.020695 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:54.020728 22171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:54.020759 22171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:54.022341 22171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:54.022661 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:54.022912 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:54.028142 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:54.028306 22171 server_base.cc:1034] running on GCE node
W20260620 00:16:54.028185 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:54.028163 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:54.028707 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:54.029038 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:54.030210 22171 hybrid_clock.cc:648] HybridClock initialized: now 1781914614030178 us; error 55 us; skew 500 ppm
Jun 20 00:16:54 dist-test-slave-28bv krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:54.032985 22171 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:54.033528 22171 webserver.cc:460] Webserver started at http://127.0.98.62:43451/ using document root <none> and password file <none>
I20260620 00:16:54.033795 22171 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:54.033866 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:54.034024 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:54.035216 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "a1cd2f5b564d4bc2955a6782d383c809"
format_stamp: "Formatted at 2026-06-20 00:16:54 on dist-test-slave-28bv"
server_key: "3d0ccc84ccd87ce3772cb50de9bbfc92"
server_key_iv: "afd3fb9ec7a5a9bb5a36f2bd7ebd9331"
server_key_version: "encryptionkey@0"
I20260620 00:16:54.035604 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1cd2f5b564d4bc2955a6782d383c809"
format_stamp: "Formatted at 2026-06-20 00:16:54 on dist-test-slave-28bv"
server_key: "3d0ccc84ccd87ce3772cb50de9bbfc92"
server_key_iv: "afd3fb9ec7a5a9bb5a36f2bd7ebd9331"
server_key_version: "encryptionkey@0"
I20260620 00:16:54.037118 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:54.037932 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:54.038146 22171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:54.038268 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "a1cd2f5b564d4bc2955a6782d383c809"
format_stamp: "Formatted at 2026-06-20 00:16:54 on dist-test-slave-28bv"
server_key: "3d0ccc84ccd87ce3772cb50de9bbfc92"
server_key_iv: "afd3fb9ec7a5a9bb5a36f2bd7ebd9331"
server_key_version: "encryptionkey@0"
I20260620 00:16:54.038360 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:54.041889 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:54.043969 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:54.044174 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:54.048425 22171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39189
I20260620 00:16:54.048463 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39189 every 8 connection(s)
I20260620 00:16:54.048902 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:54.050068 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:54.052426 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809: Bootstrap starting.
I20260620 00:16:54.053088 22239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:54.053387 22239 log.cc:824] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:54.054206 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809: No bootstrap required, opened a new log
I20260620 00:16:54.055845 22239 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } }
I20260620 00:16:54.056041 22239 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:54.056088 22239 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cd2f5b564d4bc2955a6782d383c809, State: Initialized, Role: FOLLOWER
I20260620 00:16:54.056293 22239 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } }
I20260620 00:16:54.056399 22239 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:54.056443 22239 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:54.056495 22239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:54.056869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22171
I20260620 00:16:54.056998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:54.057185 392 external_mini_cluster.cc:1428] Setting key 1726e6aee6f256c95d069f27c391d6b8
I20260620 00:16:54.057385 22239 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } }
I20260620 00:16:54.057514 22239 leader_election.cc:302] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cd2f5b564d4bc2955a6782d383c809; no voters:
I20260620 00:16:54.057700 22239 leader_election.cc:288] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:54.057864 22245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:54.057983 22239 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:54.058030 22245 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [term 1 LEADER]: Becoming Leader. State: Replica: a1cd2f5b564d4bc2955a6782d383c809, State: Running, Role: LEADER
I20260620 00:16:54.058182 22245 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } }
I20260620 00:16:54.058650 22242 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cd2f5b564d4bc2955a6782d383c809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } } }
I20260620 00:16:54.058727 22242 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:54.058965 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cd2f5b564d4bc2955a6782d383c809. Latest consensus state: current_term: 1 leader_uuid: "a1cd2f5b564d4bc2955a6782d383c809" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cd2f5b564d4bc2955a6782d383c809" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39189 } } }
I20260620 00:16:54.059145 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:54.059728 22250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:54.059837 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:54.057688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36751 (local address 127.0.98.62:39189)
0620 00:16:54.057880 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:54.057890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:54.057963 (+ 73us) server_negotiation.cc:407] Connection header received
0620 00:16:54.058104 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:54.058111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:54.058142 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:54.058595 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:54.059116 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:54.059120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:54.059120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:54.059124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:54.059162 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:54.059179 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:54.059279 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:54.059535 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 00:16:54.059589 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260620 00:16:54.060565 22250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:54.061187 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:54.062268 22250 catalog_manager.cc:1347] Generated new cluster ID: a06d6183d9c04f149eadfef7a790f690
I20260620 00:16:54.062333 22250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:54.074093 22244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:54.065026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49905 (local address 127.0.98.62:39189)
0620 00:16:54.065069 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:54.065071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:54.072878 (+ 7807us) server_negotiation.cc:407] Connection header received
0620 00:16:54.072966 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:54.072967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:54.072976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:54.073053 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:54.073763 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:54.073764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:54.073764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:54.073770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:54.073787 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:54.073792 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:54.073848 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:54.073970 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 00:16:54.073991 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:54.074416 22250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:54.074600 22250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:54.081174 22250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cd2f5b564d4bc2955a6782d383c809: Generated new TSK 0
I20260620 00:16:54.081432 22250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:54.090432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22171
2026-06-20T00:16:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:54 dist-test-slave-28bv krb5kdc[22270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:54 dist-test-slave-28bv krb5kdc[22270](info): set up 2 sockets
Jun 20 00:16:54 dist-test-slave-28bv krb5kdc[22270](info): commencing operation
krb5kdc: starting...
W20260620 00:16:55.156682 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:55.284237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:55.366487 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:55.366724 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:55.366748 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:55.368042 22283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:55.368104 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:55.368119 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:55.368139 22283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:55.368151 22283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:55.369871 22283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:55.370172 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:55.370430 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:16:55.375773 22283 server_base.cc:1034] running on GCE node
W20260620 00:16:55.375731 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:55.375823 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:55.375738 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:55.376246 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:55.376603 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:55.377831 22283 hybrid_clock.cc:648] HybridClock initialized: now 1781914615377804 us; error 37 us; skew 500 ppm
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:55.380757 22283 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:55.381378 22283 webserver.cc:460] Webserver started at http://127.0.98.62:45445/ using document root <none> and password file <none>
I20260620 00:16:55.381644 22283 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:55.381692 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:55.381850 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:55.382995 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "6e6658bb7fce48398e4af9190696300a"
format_stamp: "Formatted at 2026-06-20 00:16:55 on dist-test-slave-28bv"
server_key: "639af114c2210dedc5740fc8b3936c66"
server_key_iv: "7c94b4a72e13ad24fe856ed48bb5af37"
server_key_version: "encryptionkey@0"
I20260620 00:16:55.383379 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e6658bb7fce48398e4af9190696300a"
format_stamp: "Formatted at 2026-06-20 00:16:55 on dist-test-slave-28bv"
server_key: "639af114c2210dedc5740fc8b3936c66"
server_key_iv: "7c94b4a72e13ad24fe856ed48bb5af37"
server_key_version: "encryptionkey@0"
I20260620 00:16:55.385062 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 00:16:55.386014 22298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:55.386214 22283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:55.386325 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "6e6658bb7fce48398e4af9190696300a"
format_stamp: "Formatted at 2026-06-20 00:16:55 on dist-test-slave-28bv"
server_key: "639af114c2210dedc5740fc8b3936c66"
server_key_iv: "7c94b4a72e13ad24fe856ed48bb5af37"
server_key_version: "encryptionkey@0"
I20260620 00:16:55.386389 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:55.401777 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:55.404168 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:55.404361 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:55.408218 22283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45845
I20260620 00:16:55.408265 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45845 every 8 connection(s)
I20260620 00:16:55.408695 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:55.409499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22283
I20260620 00:16:55.409644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:55.409847 22351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:55.409942 392 external_mini_cluster.cc:1428] Setting key 49b0db3ee80b27c7ef5e25e299b9464c
I20260620 00:16:55.413383 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a: Bootstrap starting.
I20260620 00:16:55.414070 22351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:55.414388 22351 log.cc:824] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:55.415093 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a: No bootstrap required, opened a new log
I20260620 00:16:55.417402 22351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260620 00:16:55.417593 22351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:55.417641 22351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6658bb7fce48398e4af9190696300a, State: Initialized, Role: FOLLOWER
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:55.417799 22351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260620 00:16:55.417896 22351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:55.417928 22351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:55.417977 22351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:55.418669 22351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260620 00:16:55.418788 22351 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6658bb7fce48398e4af9190696300a; no voters:
I20260620 00:16:55.419003 22351 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:55.419065 22356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:55.419211 22356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [term 1 LEADER]: Becoming Leader. State: Replica: 6e6658bb7fce48398e4af9190696300a, State: Running, Role: LEADER
I20260620 00:16:55.419303 22351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:55.419306 22356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } }
I20260620 00:16:55.419801 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6658bb7fce48398e4af9190696300a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } } }
I20260620 00:16:55.419826 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6658bb7fce48398e4af9190696300a. Latest consensus state: current_term: 1 leader_uuid: "6e6658bb7fce48398e4af9190696300a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6658bb7fce48398e4af9190696300a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45845 } } }
I20260620 00:16:55.419945 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:55.419960 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:55.420148 22360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:55.420707 22360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:55.422362 22360 catalog_manager.cc:1347] Generated new cluster ID: 31e89079405141b9827ab0a226165c30
I20260620 00:16:55.422452 22360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:55.422622 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:55.410466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49741 (local address 127.0.98.62:45845)
0620 00:16:55.410700 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:55.410709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:55.410741 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:16:55.411507 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:55.411516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:55.411581 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:55.411735 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:55.412388 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:55.413809 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:55.414890 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:55.415368 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:55.415488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:55.418299 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:55.418303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:55.418303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:55.418314 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:55.420451 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:55.420922 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:55.420925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:55.420926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:55.421037 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:55.421277 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:55.421278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:55.421278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:55.421634 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:55.421806 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:55.422361 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 00:16:55.422480 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260620 00:16:55.423632 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:55.435755 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:55.425074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47129 (local address 127.0.98.62:45845)
0620 00:16:55.425111 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:55.425113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:55.429184 (+ 4071us) server_negotiation.cc:407] Connection header received
0620 00:16:55.429290 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:55.429291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:55.429302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:55.429353 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:55.429947 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:55.430479 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:55.431193 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:55.431438 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:55.431529 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:55.432886 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:55.432887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:55.432888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:55.432895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:55.434500 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:55.434946 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:55.434948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:55.434948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:55.435012 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:55.435224 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:55.435225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:55.435225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:55.435346 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:55.435457 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:55.435555 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:16:55.435639 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:16:55.440299 22360 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:55.441138 22360 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:55.452085 22360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6658bb7fce48398e4af9190696300a: Generated new TSK 0
I20260620 00:16:55.452315 22360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:55.461341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22283
2026-06-20T00:16:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22382](info): set up 2 sockets
Jun 20 00:16:55 dist-test-slave-28bv krb5kdc[22382](info): commencing operation
krb5kdc: starting...
W20260620 00:16:56.508939 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:56 dist-test-slave-28bv krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:56.644194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:56.728717 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:56.729004 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:56.729048 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:56.730342 22395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:56.730436 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:56.730461 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:56.730500 22395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:56.730557 22395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:56.732477 22395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:56.732851 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:56.733108 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:56.738828 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:56.738806 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:56.738791 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:56.739114 22395 server_base.cc:1034] running on GCE node
I20260620 00:16:56.739521 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:56.739883 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:56.741067 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781914616741046 us; error 40 us; skew 500 ppm
Jun 20 00:16:56 dist-test-slave-28bv krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:56.747327 22395 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:56.747892 22395 webserver.cc:460] Webserver started at http://127.0.98.62:33129/ using document root <none> and password file <none>
I20260620 00:16:56.748142 22395 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:56.748191 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:56.748389 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:56.749711 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "92439d86607c4679b42f669b1f02acec"
format_stamp: "Formatted at 2026-06-20 00:16:56 on dist-test-slave-28bv"
server_key: "34fb52cb1539dd8ec36f42a949148a79"
server_key_iv: "ef27094c580b24828d447d86a2bafe37"
server_key_version: "encryptionkey@0"
I20260620 00:16:56.750109 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "92439d86607c4679b42f669b1f02acec"
format_stamp: "Formatted at 2026-06-20 00:16:56 on dist-test-slave-28bv"
server_key: "34fb52cb1539dd8ec36f42a949148a79"
server_key_iv: "ef27094c580b24828d447d86a2bafe37"
server_key_version: "encryptionkey@0"
I20260620 00:16:56.751557 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:56.752686 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:56.752964 22395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:56.753123 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "92439d86607c4679b42f669b1f02acec"
format_stamp: "Formatted at 2026-06-20 00:16:56 on dist-test-slave-28bv"
server_key: "34fb52cb1539dd8ec36f42a949148a79"
server_key_iv: "ef27094c580b24828d447d86a2bafe37"
server_key_version: "encryptionkey@0"
I20260620 00:16:56.753216 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:56.781735 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:56.784353 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:56.784583 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:56.789134 22395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37979
I20260620 00:16:56.789168 22462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37979 every 8 connection(s)
I20260620 00:16:56.789686 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:56.789863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22395
I20260620 00:16:56.789994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:56.790381 392 external_mini_cluster.cc:1428] Setting key 1ed178e13f13f7a4e9456883633ea053
I20260620 00:16:56.791015 22463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:56.794675 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec: Bootstrap starting.
I20260620 00:16:56.795539 22463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:56.795903 22463 log.cc:824] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:56.796723 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec: No bootstrap required, opened a new log
I20260620 00:16:56.797160 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:56.791090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53895 (local address 127.0.98.62:37979)
0620 00:16:56.791328 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:56.791339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:56.791374 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:16:56.791961 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:56.791972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:56.792041 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:56.792205 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:56.792439 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:56.793933 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:56.794772 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:56.795414 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:56.795559 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:56.796425 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:56.796430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:56.796430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:56.796680 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:56.796712 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:56.796724 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:56.796798 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:56.796887 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 00:16:56.796958 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
I20260620 00:16:56.798195 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:56.799171 22463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } }
I20260620 00:16:56.799361 22463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:56.799410 22463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92439d86607c4679b42f669b1f02acec, State: Initialized, Role: FOLLOWER
I20260620 00:16:56.799547 22463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } }
I20260620 00:16:56.799736 22463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:56.799799 22463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:56.799858 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:56.800746 22463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } }
I20260620 00:16:56.800863 22463 leader_election.cc:302] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92439d86607c4679b42f669b1f02acec; no voters:
I20260620 00:16:56.801060 22463 leader_election.cc:288] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:56.801155 22473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:56.801357 22463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:56.801400 22473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [term 1 LEADER]: Becoming Leader. State: Replica: 92439d86607c4679b42f669b1f02acec, State: Running, Role: LEADER
I20260620 00:16:56.801586 22473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } }
I20260620 00:16:56.801987 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92439d86607c4679b42f669b1f02acec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } } }
I20260620 00:16:56.802073 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:56.802256 22475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92439d86607c4679b42f669b1f02acec. Latest consensus state: current_term: 1 leader_uuid: "92439d86607c4679b42f669b1f02acec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92439d86607c4679b42f669b1f02acec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37979 } } }
I20260620 00:16:56.802338 22475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:56.802901 22480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:56.803504 22480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:56.804003 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:56.799675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47903 (local address 127.0.98.62:37979)
0620 00:16:56.799713 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:56.799715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:56.799873 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:16:56.799936 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:56.799937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:56.799950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:56.800021 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:56.800114 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:56.800891 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:56.802112 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:56.802467 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:56.802514 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:56.803574 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:56.803576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:56.803576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:56.803582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:56.803601 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:56.803606 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:56.803682 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:56.803777 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:16:56.803917 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:16:56.805420 22480 catalog_manager.cc:1347] Generated new cluster ID: 03a988572ce548a1ad0f7f7cff0b908f
I20260620 00:16:56.805480 22480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:56.828094 22480 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:56.828919 22480 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:56.834942 22480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92439d86607c4679b42f669b1f02acec: Generated new TSK 0
I20260620 00:16:56.835196 22480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:56.847961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22395
2026-06-20T00:16:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:56 dist-test-slave-28bv krb5kdc[22494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:56 dist-test-slave-28bv krb5kdc[22494](info): set up 2 sockets
Jun 20 00:16:56 dist-test-slave-28bv krb5kdc[22494](info): commencing operation
krb5kdc: starting...
W20260620 00:16:57.945212 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:58 dist-test-slave-28bv krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:58.085223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:58.169050 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:58.169433 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:58.169469 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:58.171295 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:58.171370 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:58.171396 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:58.171414 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:58.171442 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:58.173691 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:58.174034 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:58.174289 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:58.179479 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:58.179483 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:58.179510 22507 server_base.cc:1034] running on GCE node
W20260620 00:16:58.179479 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:58.180131 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:58.180467 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:58.181669 22507 hybrid_clock.cc:648] HybridClock initialized: now 1781914618181628 us; error 57 us; skew 500 ppm
Jun 20 00:16:58 dist-test-slave-28bv krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:58.184528 22507 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:58.185101 22507 webserver.cc:460] Webserver started at http://127.0.98.62:42257/ using document root <none> and password file <none>
I20260620 00:16:58.185387 22507 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:58.185467 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:58.185635 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:58.186827 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "c938aa3f09ba423aa311b1ef199c9fd8"
format_stamp: "Formatted at 2026-06-20 00:16:58 on dist-test-slave-28bv"
server_key: "f3e55c0681be75158b97eb2ae5edf832"
server_key_iv: "02d64828dc6bb8aca051553290b3a2d6"
server_key_version: "encryptionkey@0"
I20260620 00:16:58.187254 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "c938aa3f09ba423aa311b1ef199c9fd8"
format_stamp: "Formatted at 2026-06-20 00:16:58 on dist-test-slave-28bv"
server_key: "f3e55c0681be75158b97eb2ae5edf832"
server_key_iv: "02d64828dc6bb8aca051553290b3a2d6"
server_key_version: "encryptionkey@0"
I20260620 00:16:58.188948 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 00:16:58.189844 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:58.190083 22507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:58.190212 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "c938aa3f09ba423aa311b1ef199c9fd8"
format_stamp: "Formatted at 2026-06-20 00:16:58 on dist-test-slave-28bv"
server_key: "f3e55c0681be75158b97eb2ae5edf832"
server_key_iv: "02d64828dc6bb8aca051553290b3a2d6"
server_key_version: "encryptionkey@0"
I20260620 00:16:58.190312 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:58.193946 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:58.196662 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:58.196900 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:58.201207 22507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35023
I20260620 00:16:58.201298 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35023 every 8 connection(s)
I20260620 00:16:58.201645 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:58.202772 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:58.205309 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: Bootstrap starting.
I20260620 00:16:58.205945 22575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:58.206247 22575 log.cc:824] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:58.207002 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: No bootstrap required, opened a new log
I20260620 00:16:58.208796 22575 raft_consensus.cc:348] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } }
I20260620 00:16:58.208952 22575 raft_consensus.cc:374] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:16:58.208976 22575 raft_consensus.cc:729] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c938aa3f09ba423aa311b1ef199c9fd8, State: Initialized, Role: FOLLOWER
I20260620 00:16:58.209148 22575 consensus_queue.cc:260] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } }
I20260620 00:16:58.209215 22575 raft_consensus.cc:388] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:16:58.209237 22575 raft_consensus.cc:482] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:16:58.209266 22575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:58.209904 22575 raft_consensus.cc:504] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } }
I20260620 00:16:58.210026 22575 leader_election.cc:302] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c938aa3f09ba423aa311b1ef199c9fd8; no voters:
I20260620 00:16:58.210253 22575 leader_election.cc:288] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:16:58.210353 22578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:58.210513 22575 sys_catalog.cc:564] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:58.210803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22507
I20260620 00:16:58.210942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:58.211128 392 external_mini_cluster.cc:1428] Setting key d9cf762cab945f3fa1bdc100cfc7d218
I20260620 00:16:58.210531 22578 raft_consensus.cc:686] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [term 1 LEADER]: Becoming Leader. State: Replica: c938aa3f09ba423aa311b1ef199c9fd8, State: Running, Role: LEADER
I20260620 00:16:58.211341 22578 consensus_queue.cc:237] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } }
I20260620 00:16:58.212095 22580 sys_catalog.cc:455] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c938aa3f09ba423aa311b1ef199c9fd8. Latest consensus state: current_term: 1 leader_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } }
I20260620 00:16:58.212433 22580 sys_catalog.cc:458] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:58.212270 22579 sys_catalog.cc:455] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c938aa3f09ba423aa311b1ef199c9fd8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35023 } } }
I20260620 00:16:58.212649 22579 sys_catalog.cc:458] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8 [sys.catalog]: This master's current role is: LEADER
W20260620 00:16:58.213096 22594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:58.213156 22594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:58.213219 22596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:58.213406 22596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:58.214042 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:58.211776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46335 (local address 127.0.98.62:35023)
0620 00:16:58.212005 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:58.212015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:58.212050 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:16:58.212518 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:58.212528 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:58.212596 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:58.212748 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:58.213548 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:58.213551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:58.213551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:58.213556 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:58.213606 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:58.213620 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:58.213702 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:58.213814 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 00:16:58.213856 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":126,"threads_started":1}
I20260620 00:16:58.215119 22596 catalog_manager.cc:1347] Generated new cluster ID: 06fe66a45c664690b58c363ff9084854
I20260620 00:16:58.215170 22596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:58.215340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:58.218006 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:58.216790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.0.98.62:35023)
0620 00:16:58.216834 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:58.216837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:58.216975 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:16:58.217038 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:58.217039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:58.217045 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:58.217090 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:58.217739 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:58.217741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:58.217741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:16:58.217746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:58.217763 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:16:58.217767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:58.217826 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:58.217887 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:16:58.217903 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:58.234385 22596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:16:58.234632 22596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:16:58.242254 22596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c938aa3f09ba423aa311b1ef199c9fd8: Generated new TSK 0
I20260620 00:16:58.242506 22596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:16:58.247841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22507
2026-06-20T00:16:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1399 ms)
[----------] 18 tests from AuthTokenIssuingTest (24528 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:16:58 dist-test-slave-28bv krb5kdc[22606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:16:58 dist-test-slave-28bv krb5kdc[22606](info): set up 2 sockets
Jun 20 00:16:58 dist-test-slave-28bv krb5kdc[22606](info): commencing operation
krb5kdc: starting...
W20260620 00:16:59.317361 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:16:59.452775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:16:59.546682 22619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:16:59.546949 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:59.546968 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:59.546991 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:59.548264 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:59.548333 22619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:59.548360 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:59.548374 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:59.548386 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:59.550310 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:59.550572 22619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:59.550793 22619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:59.556071 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.556254 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.556071 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:59.556509 22619 server_base.cc:1034] running on GCE node
I20260620 00:16:59.556859 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:59.557253 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:59.558436 22619 hybrid_clock.cc:648] HybridClock initialized: now 1781914619558363 us; error 83 us; skew 500 ppm
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:59.561740 22619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:16:59.562302 22619 webserver.cc:460] Webserver started at http://127.0.98.62:45167/ using document root <none> and password file <none>
I20260620 00:16:59.562600 22619 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:59.562680 22619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:59.562849 22619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:59.564097 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "4f2e286ad6944765b83b79fcfe797ea8"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "40684861fb0508867d5ffbf13b7af50a"
server_key_iv: "2b180df3240b575b5cfc6287fb8ce494"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.564625 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f2e286ad6944765b83b79fcfe797ea8"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "40684861fb0508867d5ffbf13b7af50a"
server_key_iv: "2b180df3240b575b5cfc6287fb8ce494"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.566190 22619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:59.567224 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:59.567456 22619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:16:59.567586 22619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "4f2e286ad6944765b83b79fcfe797ea8"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "40684861fb0508867d5ffbf13b7af50a"
server_key_iv: "2b180df3240b575b5cfc6287fb8ce494"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.567684 22619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:59.583439 22619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:59.586369 22619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:59.586570 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:59.591257 22619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46071
I20260620 00:16:59.591295 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46071 every 8 connection(s)
I20260620 00:16:59.591720 22619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:16:59.592887 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:59.594544 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.599268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22619
I20260620 00:16:59.599421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:16:59.599638 392 external_mini_cluster.cc:1428] Setting key 6a42624bd12f22ac5775d1db1150df20
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:59.605844 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.595315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.0.1:49180)
0620 00:16:59.595492 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.595531 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.596023 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.596407 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.596410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.596839 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.597005 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.597010 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.598120 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.598121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.598816 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.598818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.599421 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.599422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.599492 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.600088 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.600096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.601674 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.604050 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.604057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.604060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.604291 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.604653 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.604655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.604655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.604752 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.605272 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.605278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.605346 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.605422 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.605592 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260620 00:16:59.605856 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.595517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.0.98.62:46071)
0620 00:16:59.595708 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.595721 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.595752 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:16:59.596230 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.596232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.596253 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.596348 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.597148 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.597982 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.598918 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.599232 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.599429 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.601785 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.601794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.601795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.601804 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.603939 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.604387 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.604391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.604391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.604529 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.604832 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.604833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.604833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.605188 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.605300 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.605502 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.605682 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":155,"thread_start_us":129,"threads_started":1}
I20260620 00:16:59.606761 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.607158 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.607033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44638)
0620 00:16:59.607085 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.607133 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:16:59.607296 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
I20260620 00:16:59.607743 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.607648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44648)
0620 00:16:59.607693 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.607718 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260620 00:16:59.607839 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44419: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:16:59.628859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44419
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:16:59.657070 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } attempt: 1
I20260620 00:16:59.657629 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.657483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44650)
0620 00:16:59.657556 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.657598 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 00:16:59.658046 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.657944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44656)
0620 00:16:59.657994 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.658019 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:16:59.658138 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44419: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
W20260620 00:16:59.713191 22694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:16:59.713477 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:59.713510 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:59.713555 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:59.714890 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:59.714978 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:59.715003 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:59.715026 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:59.715055 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:59.716732 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:59.717043 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:59.717295 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:59.722941 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.722954 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.723012 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:59.723217 22694 server_base.cc:1034] running on GCE node
I20260620 00:16:59.723562 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:59.723903 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:59.725119 22694 hybrid_clock.cc:648] HybridClock initialized: now 1781914619725111 us; error 56 us; skew 500 ppm
I20260620 00:16:59.726275 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } attempt: 2
I20260620 00:16:59.726706 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.726534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44672)
0620 00:16:59.726647 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.726677 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:16:59.727035 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.726943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44682)
0620 00:16:59.726987 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.727011 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:16:59.727120 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44419: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44419: connect: Connection refused (error 111)
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:59.728269 22694 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:16:59.728861 22694 webserver.cc:460] Webserver started at http://127.0.98.61:38583/ using document root <none> and password file <none>
I20260620 00:16:59.729146 22694 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:59.729224 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:59.729398 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:59.730629 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data/instance:
uuid: "aeab81a4d75948fc8ee703d59799fe0f"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "dc8a951b8b98cde2fe62137adcab9ba5"
server_key_iv: "47bfc8e822a61647794c52e79616e078"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.731038 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal/instance:
uuid: "aeab81a4d75948fc8ee703d59799fe0f"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "dc8a951b8b98cde2fe62137adcab9ba5"
server_key_iv: "47bfc8e822a61647794c52e79616e078"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.732590 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:59.733522 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:59.733820 22694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:16:59.733940 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "aeab81a4d75948fc8ee703d59799fe0f"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "dc8a951b8b98cde2fe62137adcab9ba5"
server_key_iv: "47bfc8e822a61647794c52e79616e078"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.734030 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:59.744693 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:59.746840 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:59.747017 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:59.751484 22694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44419
I20260620 00:16:59.751564 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44419 every 8 connection(s)
I20260620 00:16:59.751946 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:16:59.753059 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:59.754159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22694
I20260620 00:16:59.754245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-1/wal/instance
I20260620 00:16:59.754436 392 external_mini_cluster.cc:1428] Setting key f6a0bf31a1b2e7c8d4483950f681b18f
I20260620 00:16:59.754774 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 00:16:59.765893 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.755534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.98.62:46071)
0620 00:16:59.755592 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.755594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.755812 (+ 218us) server_negotiation.cc:407] Connection header received
0620 00:16:59.756321 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.756322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.756334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.756392 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.757511 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.758068 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.759210 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.759444 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.759537 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.762300 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.762302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.762302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.762312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.764142 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.764645 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.764646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.764647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.764734 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.765100 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.765101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.765101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.765291 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.765392 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.765699 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.765795 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:16:59.766108 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.755490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.0.1:49196)
0620 00:16:59.755664 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.755716 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.756217 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.756465 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.756470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.756951 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.757161 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.757166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.758230 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.758231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.759080 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.759083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.759566 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.759567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.759704 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.760355 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.760365 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.762138 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.764242 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.764254 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.764258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.764525 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.764806 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.764807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.764808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.764962 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.765383 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.765391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.765555 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.765666 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.765857 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260620 00:16:59.766889 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:16:59.776067 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.767291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44690)
0620 00:16:59.767356 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.767367 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.767411 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.767738 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.767739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.767884 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.767980 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.767981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.768966 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.768967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.769481 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.769482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.770279 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.770280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.770374 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.770980 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.770987 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.772458 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.774569 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.774571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.774571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.774842 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.775144 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.775145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.775145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.775214 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.775588 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.775588 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.775629 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.775725 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.775977 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":126}
I20260620 00:16:59.776074 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.767367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.0.98.61:44419)
0620 00:16:59.767547 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.767551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.767568 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:16:59.767584 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.767586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.767597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.767694 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.768065 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.768868 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.769595 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.770050 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.770243 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.772592 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.772606 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.772611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.772618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.774374 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.774958 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.774962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.774963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.775056 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.775316 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.775317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.775317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.775506 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.775654 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.775771 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.775920 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
I20260620 00:16:59.777207 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.777704 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.777509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47736)
0620 00:16:59.777618 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.777674 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 00:16:59.777928 22711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
I20260620 00:16:59.778651 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.778510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47738)
0620 00:16:59.778584 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.778627 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 00:16:59.778832 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41721: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:16:59.786160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41721
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:16:59.818298 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } attempt: 3
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:16:59.826462 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.819026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.0.98.61:44419)
0620 00:16:59.819166 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.819168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.819182 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:16:59.819287 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.819287 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.819298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.819372 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.819703 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.820270 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.821081 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.821302 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.821384 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.823471 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.823473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.823473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.823481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.825057 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.825523 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.825524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.825524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.825589 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.825862 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.825863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.825863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.826017 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.826087 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.826311 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.826392 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 00:16:59.826486 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.818877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44692)
0620 00:16:59.818986 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.819007 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.819169 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.819379 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.819380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.819553 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.819617 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.819619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.820360 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.820362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.820971 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.820973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.821382 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.821383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.821463 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.821945 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.821957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.823363 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.825149 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.825151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.825151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.825418 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.825675 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.825676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.825676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.825751 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.826096 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.826097 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.826160 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.826253 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.826372 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 00:16:59.826998 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.827796 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.827569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47742)
0620 00:16:59.827696 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.827748 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 00:16:59.828047 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } attempt: 1
I20260620 00:16:59.828632 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.828298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47752)
0620 00:16:59.828527 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.828590 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260620 00:16:59.828668 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.828449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47768)
0620 00:16:59.828534 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.828626 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 00:16:59.828738 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41721: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
I20260620 00:16:59.829036 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.828952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47778)
0620 00:16:59.828984 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.829010 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 00:16:59.829144 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41721: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
I20260620 00:16:59.860054 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } attempt: 1
I20260620 00:16:59.860808 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.860588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47780)
0620 00:16:59.860647 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.860760 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 00:16:59.861311 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.861202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47790)
0620 00:16:59.861250 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.861286 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:16:59.861474 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41721: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
W20260620 00:16:59.879952 22769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:16:59.880264 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:16:59.880285 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:16:59.880311 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:16:59.881916 22769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:16:59.881984 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:16:59.882000 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:16:59.882015 22769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:16:59.882028 22769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:16:59.884068 22769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46071,127.0.98.61:44419,127.0.98.60:41721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:16:59.884424 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:16:59.884770 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:16:59.890417 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.890458 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:16:59.890461 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:16:59.891024 22769 server_base.cc:1034] running on GCE node
I20260620 00:16:59.891346 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:16:59.891692 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:16:59.892932 22769 hybrid_clock.cc:648] HybridClock initialized: now 1781914619892904 us; error 52 us; skew 500 ppm
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:16:59.895722 22769 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:16:59.896330 22769 webserver.cc:460] Webserver started at http://127.0.98.60:37699/ using document root <none> and password file <none>
I20260620 00:16:59.896656 22769 fs_manager.cc:359] Metadata directory not provided
I20260620 00:16:59.896732 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:16:59.896910 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:16:59.897298 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } attempt: 2
I20260620 00:16:59.897737 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.897592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47800)
0620 00:16:59.897667 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.897707 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 00:16:59.898072 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.897961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47804)
0620 00:16:59.898006 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.898030 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:16:59.898157 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41721: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41721: connect: Connection refused (error 111)
I20260620 00:16:59.898252 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data/instance:
uuid: "5d5285f652d04233b017179efe4d40b6"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "a3da4aafc675045f94644a44a2819786"
server_key_iv: "3b99d7cc6f915625b5afe6c584f180ae"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.898649 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal/instance:
uuid: "5d5285f652d04233b017179efe4d40b6"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "a3da4aafc675045f94644a44a2819786"
server_key_iv: "3b99d7cc6f915625b5afe6c584f180ae"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.900151 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:16:59.901093 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:16:59.901410 22769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:16:59.901496 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "5d5285f652d04233b017179efe4d40b6"
format_stamp: "Formatted at 2026-06-20 00:16:59 on dist-test-slave-28bv"
server_key: "a3da4aafc675045f94644a44a2819786"
server_key_iv: "3b99d7cc6f915625b5afe6c584f180ae"
server_key_version: "encryptionkey@0"
I20260620 00:16:59.901655 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:16:59.909991 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:16:59.912438 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:16:59.912649 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:16:59.916993 22769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41721
I20260620 00:16:59.917068 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41721 every 8 connection(s)
I20260620 00:16:59.917428 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:16:59.918473 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:16:59.920173 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.921700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 22769
I20260620 00:16:59.921802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914322045185-392-0/minicluster-data/master-2/wal/instance
I20260620 00:16:59.922034 392 external_mini_cluster.cc:1428] Setting key 89f06085ec5f2e75be4e606e88abbdac
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:16:59.932099 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.920962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.98.62:46071)
0620 00:16:59.921037 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.921039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.921316 (+ 277us) server_negotiation.cc:407] Connection header received
0620 00:16:59.921977 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.921978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.921990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.922054 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.923502 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.924062 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.925554 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.925792 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.925872 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.928633 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.928635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.928636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.928645 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.930336 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.930940 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.930941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.930942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.930998 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.931328 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.931330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.931330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.931471 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.931568 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.931904 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.931994 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:16:59.932428 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.922715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.0.98.60:41721)
0620 00:16:59.922953 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.922959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.922984 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:16:59.923010 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.923012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.923031 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.923128 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.923763 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.924690 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.925446 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.925795 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.925917 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.928958 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.928967 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.928967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.928975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.930974 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.931352 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.931355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.931355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.931421 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.931641 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.931642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.931642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.931869 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.932074 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.932129 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.932214 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":155,"threads_started":1}
I20260620 00:16:59.932533 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.920946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46071 (local address 127.0.0.1:49200)
0620 00:16:59.921172 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.921212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.921855 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.922133 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.922302 (+ 169us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.922732 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.923332 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.923339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.924209 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.924210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.925423 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.925425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.925919 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.925921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.925988 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.926660 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.926667 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.928402 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.930440 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.930446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.930448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.930849 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.931083 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.931084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.931085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.931221 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.931561 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.931568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.931730 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.931866 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.932038 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260620 00:16:59.933157 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.936615 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } attempt: 2
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:16:59.941394 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.933394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44419 (local address 127.0.0.1:44696)
0620 00:16:59.933465 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.933477 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.933517 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.933634 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.933635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.933808 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.933858 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.933861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.934619 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.934620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.935278 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.935281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.935620 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.935621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.935681 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.936239 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.936244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.938057 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.940127 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.940128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.940128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.940373 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.940642 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.940643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.940644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.940711 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.941058 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.941058 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.941108 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.941178 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.941292 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 00:16:59.941394 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.933471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.0.98.61:44419)
0620 00:16:59.933523 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.933525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.933536 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:16:59.933566 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.933567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.933577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.933622 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.933947 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.934531 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.935364 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.935550 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.935602 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.938210 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.938212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.938212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.938221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.940016 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.940459 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.940461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.940461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.940530 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.940779 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.940780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.940781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.940958 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.941005 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.941208 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.941291 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:16:59.941731 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:16:59.942287 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.933435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.0.98.62:46071)
0620 00:16:59.933471 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.933473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.933748 (+ 275us) server_negotiation.cc:407] Connection header received
0620 00:16:59.933811 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.933811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.933820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.933867 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.934347 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.934913 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.935766 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.935940 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.935992 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.938671 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.938673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.938673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.938680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.940589 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.941003 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.941004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.941004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.941071 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.941297 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.941298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.941298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.941427 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.941604 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.942117 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.942195 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:16:59.949412 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.938369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.0.98.60:41721)
0620 00:16:59.938419 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.938421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.938429 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:16:59.938453 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.938453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.938465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.939684 (+ 1219us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.940090 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.940790 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.941982 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.942318 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.942738 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.945663 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.945664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.945665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.945671 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.947548 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.948230 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.948231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.948232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.948300 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.948610 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.948611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.948611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.948851 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.948961 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.949175 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.949279 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:16:59.949445 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.936890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47816)
0620 00:16:59.938201 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.938213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.938262 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.939601 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.939602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.939921 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.939986 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.939988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.940871 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.940872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.941654 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.941655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.942730 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.942731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.942793 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.943225 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.943230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.944709 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.947675 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.947676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.947677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.948117 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.948407 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.948408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.948408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.948480 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.948962 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.948963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.949029 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.949125 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.949302 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283}
I20260620 00:16:59.951849 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: Bootstrap starting.
I20260620 00:16:59.952801 22687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:59.952849 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.942005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47824)
0620 00:16:59.942277 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.942289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.942341 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.942467 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.942468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.942613 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.942640 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.942663 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.944404 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.944405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.945092 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.945095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.945735 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.945736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.945804 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.946444 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.946450 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.948920 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.951405 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.951406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.951406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.951586 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.951811 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.951812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.951812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.951869 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.952448 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:16:59.952449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:16:59.952498 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:16:59.952606 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:16:59.952736 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260620 00:16:59.953012 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.942088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47824 (local address 127.0.98.60:41721)
0620 00:16:59.942285 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.942287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.942325 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:16:59.942396 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.942397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.942408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.942467 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.942964 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.944316 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.945177 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.945539 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.945635 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.949017 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.949018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.949019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.949026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.951282 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.951676 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.951677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.951677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.951741 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.952014 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.952014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.952015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.952227 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.952307 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.952637 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.952803 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":179,"thread_start_us":155,"threads_started":1}
I20260620 00:16:59.953169 22687 log.cc:824] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:59.953704 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.943024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.0.98.61:44419)
0620 00:16:59.943061 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.943063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.943072 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:16:59.943130 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.943131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.943139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.943188 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.944042 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.944671 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.945732 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.946065 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.946168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.949431 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.949433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.949433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.949443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.951739 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.952181 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.952183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.952183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.952246 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.952487 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.952488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.952489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.952662 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.952848 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.953523 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.953618 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:16:59.954103 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: No bootstrap required, opened a new log
I20260620 00:16:59.954761 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: Bootstrap starting.
I20260620 00:16:59.955585 22837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: Neither blocks nor log segments found. Creating new log.
I20260620 00:16:59.956087 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:16:59.956095 22837 log.cc:824] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:16:59.956678 22687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.956854 22687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2e286ad6944765b83b79fcfe797ea8, State: Initialized, Role: FOLLOWER
I20260620 00:16:59.956951 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: No bootstrap required, opened a new log
I20260620 00:16:59.956934 22687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.957304 22687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:59.957281 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.957368 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:16:59.957414 22661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4f2e286ad6944765b83b79fcfe797ea8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210
I20260620 00:16:59.957526 22661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:16:59.957564 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:59.958580 22661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.958892 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.959174 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d5285f652d04233b017179efe4d40b6, State: Initialized, Role: FOLLOWER
W20260620 00:16:59.959542 22856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:59.959595 22856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:59.958961 22661 leader_election.cc:288] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [CANDIDATE]: Term 1 election: Requested vote from peers aeab81a4d75948fc8ee703d59799fe0f (127.0.98.61:44419), 5d5285f652d04233b017179efe4d40b6 (127.0.98.60:41721)
I20260620 00:16:59.959573 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.960223 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f2e286ad6944765b83b79fcfe797ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d5285f652d04233b017179efe4d40b6"
I20260620 00:16:59.960333 22811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:16:59.961277 22811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2e286ad6944765b83b79fcfe797ea8 in term 1.
I20260620 00:16:59.961555 22636 leader_election.cc:302] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2e286ad6944765b83b79fcfe797ea8, 5d5285f652d04233b017179efe4d40b6; no voters:
I20260620 00:16:59.961696 22845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:16:59.961738 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:16:59.961858 22845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2e286ad6944765b83b79fcfe797ea8, State: Running, Role: LEADER
I20260620 00:16:59.961971 22845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:16:59.962491 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2e286ad6944765b83b79fcfe797ea8. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.962587 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:59.961413 22619 master_runner.cc:186] Error getting master registration for 127.0.98.61:44419: OK, instance_id { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" instance_seqno: 1781914619750216 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:16:59.964015 22619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 00:16:59.965245 22879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:16:59.965392 22879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:16:59.966301 22769 master_runner.cc:186] Error getting master registration for 127.0.98.61:44419: OK, instance_id { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" instance_seqno: 1781914619750216 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:16:59.969007 22769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:16:59.972533 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.960994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.0.98.62:46071)
0620 00:16:59.961040 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.961042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.964241 (+ 3199us) server_negotiation.cc:407] Connection header received
0620 00:16:59.964347 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.964348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.964359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.964390 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.965324 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.966157 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.967208 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.967513 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.967559 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.969343 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.969345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.969345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.969354 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.971179 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.971617 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.971618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.971619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.971693 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.971900 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.971901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.971901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.972032 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.972116 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.972331 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.972415 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:16:59.972685 22871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:16:59.972973 22871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:16:59.975042 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [term 1 FOLLOWER]: Refusing update from remote peer 4f2e286ad6944765b83b79fcfe797ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:16:59.975481 22866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:16:59.978232 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.961349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.0.98.61:44419)
0620 00:16:59.961391 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.961393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.969460 (+ 8067us) server_negotiation.cc:407] Connection header received
0620 00:16:59.969588 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.969589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.969603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.969651 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.970363 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.971134 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.972603 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.972927 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.972973 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.974533 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.974535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.974535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.974543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.976645 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.977055 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.977056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.977057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.977134 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.977648 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.977650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.977650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.977801 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.977921 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.978041 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.978126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:16:59.978747 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2e286ad6944765b83b79fcfe797ea8. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.978940 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:16:59.979362 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.961135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.98.60:41721)
0620 00:16:59.964122 (+ 2987us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.964125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.964161 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:16:59.964190 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.964191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.964229 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.966465 (+ 2236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.966534 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.967763 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.969099 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.969623 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.971517 (+ 1894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.971647 (+ 130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.971648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.971648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.971653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.975501 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.976419 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.976420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.976420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.976511 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.976810 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.976811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.976811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.977054 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:16:59.979107 (+ 2053us) server_negotiation.cc:1035] Waiting for connection context
0620 00:16:59.979174 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:16:59.979271 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2968}
I20260620 00:16:59.979839 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.979948 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:16:59.980011 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.980137 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:16:59.980475 22883 mvcc.cc:205] Tried to move back new op lower bound from 7298722283414134784 to 7298722283365113856. Current Snapshot: MvccSnapshot[applied={T|T < 7298722283414134784}]
I20260620 00:16:59.980589 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:16:59.980660 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:16:59.985510 22871 catalog_manager.cc:1347] Generated new cluster ID: a76e0569cde040bea04d59b217cdc1b2
I20260620 00:16:59.985601 22871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:16:59.989287 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } attempt: 3
Jun 20 00:16:59 dist-test-slave-28bv krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:17:00.001262 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.989661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41721 (local address 127.0.0.1:47850)
0620 00:16:59.989787 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 00:16:59.989799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:16:59.989846 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:16:59.990154 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:16:59.990156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:16:59.990375 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:16:59.990427 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.990429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.991440 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.991441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.992282 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:16:59.992286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.993241 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.993243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:16:59.993328 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.993938 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:16:59.993944 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:16:59.995633 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:16:59.998857 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.998858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.998859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.999155 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:16:59.999736 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:16:59.999737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:16:59.999737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:16:59.999839 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:00.000560 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:00.000562 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:00.000935 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 00:17:00.001003 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:17:00.001145 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 00:17:00.001968 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:16:59.989926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.98.60:41721)
0620 00:16:59.989981 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:16:59.989983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:16:59.989991 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:16:59.990016 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:16:59.990017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:16:59.990027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:16:59.990083 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:16:59.990534 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.991314 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.992403 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:16:59.993076 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:16:59.993571 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:16:59.995793 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:16:59.995795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:16:59.995795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:16:59.995834 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:16:59.998643 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.999318 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.999319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.999319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:16:59.999591 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:16:59.999936 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:16:59.999937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:16:59.999937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:00.000364 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:00.001522 (+ 1158us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:00.001591 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:17:00.001825 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:00.004220 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: Bootstrap starting.
I20260620 00:17:00.005306 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:00.005702 22762 log.cc:824] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:00.008189 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: No bootstrap required, opened a new log
I20260620 00:17:00.008200 22871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:00.009116 22871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:17:00.010762 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:17:00.011011 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeab81a4d75948fc8ee703d59799fe0f, State: Initialized, Role: FOLLOWER
I20260620 00:17:00.011121 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } }
I20260620 00:17:00.011546 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:00.011628 22735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:00.012889 22735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [term 1 FOLLOWER]: Refusing update from remote peer 4f2e286ad6944765b83b79fcfe797ea8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:00.013584 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4f2e286ad6944765b83b79fcfe797ea8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aeab81a4d75948fc8ee703d59799fe0f"
I20260620 00:17:00.014629 22736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2e286ad6944765b83b79fcfe797ea8 in term 1.
I20260620 00:17:00.015200 22845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:17:00.016763 22899 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:17:00.016839 22899 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:17:00.011508 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:17:00.020159 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:00.020282 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f2e286ad6944765b83b79fcfe797ea8. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:17:00.020561 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:00.020658 22888 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:17:00.020741 22888 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:00.022316 22902 mvcc.cc:205] Tried to move back new op lower bound from 7298722283543646208 to 7298722283365113856. Current Snapshot: MvccSnapshot[applied={T|T < 7298722283414134784}]
I20260620 00:17:00.026988 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f2e286ad6944765b83b79fcfe797ea8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2e286ad6944765b83b79fcfe797ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46071 } } peers { permanent_uuid: "aeab81a4d75948fc8ee703d59799fe0f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44419 } } peers { permanent_uuid: "5d5285f652d04233b017179efe4d40b6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41721 } } }
I20260620 00:17:00.027275 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:00.029263 22871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f2e286ad6944765b83b79fcfe797ea8: Generated new TSK 0
I20260620 00:17:00.029577 22871 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:17:00.061894 22690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.059859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.0.98.62:46071)
0620 00:17:00.059922 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.059925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.061149 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 00:17:00.061262 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.061263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.061276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.061308 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.061816 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49216: BlockingRecv error: recv got EOF from 127.0.0.1:49216 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 00:17:00.062717 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.060291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.0.98.61:44419)
0620 00:17:00.060333 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.060335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.062131 (+ 1796us) server_negotiation.cc:407] Connection header received
0620 00:17:00.062234 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.062235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.062246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.062277 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.062653 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44718: BlockingRecv error: recv got EOF from 127.0.0.1:44718 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:17:00.063524 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.060677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.0.98.60:41721)
0620 00:17:00.060732 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.060734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.062975 (+ 2241us) server_negotiation.cc:407] Connection header received
0620 00:17:00.063070 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.063071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.063082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.063111 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.063459 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47866: BlockingRecv error: recv got EOF from 127.0.0.1:47866 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:17:00.070008 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.068976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.0.98.61:44419)
0620 00:17:00.069028 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.069031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.069296 (+ 265us) server_negotiation.cc:407] Connection header received
0620 00:17:00.069428 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.069430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.069453 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.069486 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.069941 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44732: BlockingRecv error: recv got EOF from 127.0.0.1:44732 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:17:00.077483 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.072436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.0.98.61:44419)
0620 00:17:00.072474 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.072476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.076804 (+ 4328us) server_negotiation.cc:407] Connection header received
0620 00:17:00.076937 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.076938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.076951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.076982 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.077412 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44736: BlockingRecv error: recv got EOF from 127.0.0.1:44736 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:17:00.081879 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.080923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.0.98.61:44419)
0620 00:17:00.080981 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.080984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.081288 (+ 304us) server_negotiation.cc:407] Connection header received
0620 00:17:00.081390 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.081391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.081403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.081432 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.081814 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44744: BlockingRecv error: recv got EOF from 127.0.0.1:44744 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:17:00.088201 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.087187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.0.98.61:44419)
0620 00:17:00.087246 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.087248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.087564 (+ 316us) server_negotiation.cc:407] Connection header received
0620 00:17:00.087677 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.087678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.087690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.087722 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.088136 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44748: BlockingRecv error: recv got EOF from 127.0.0.1:44748 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:17:00.098719 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.097688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.61:44419)
0620 00:17:00.097735 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.097737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.098048 (+ 311us) server_negotiation.cc:407] Connection header received
0620 00:17:00.098165 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.098166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.098178 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.098210 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.098648 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44762: BlockingRecv error: recv got EOF from 127.0.0.1:44762 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:17:00.117298 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.116267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.98.61:44419)
0620 00:17:00.116313 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.116315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.116654 (+ 339us) server_negotiation.cc:407] Connection header received
0620 00:17:00.116767 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.116769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.116782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.116813 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.117220 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:17:00.151996 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.151015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.0.98.61:44419)
0620 00:17:00.151063 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.151066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.151375 (+ 309us) server_negotiation.cc:407] Connection header received
0620 00:17:00.151488 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.151489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.151501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.151533 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.151931 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44786: BlockingRecv error: recv got EOF from 127.0.0.1:44786 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:17:00.218515 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.217494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.98.61:44419)
0620 00:17:00.217548 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.217551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.217780 (+ 229us) server_negotiation.cc:407] Connection header received
0620 00:17:00.217834 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.217835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.217849 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.217920 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.218450 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44792: BlockingRecv error: recv got EOF from 127.0.0.1:44792 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 00:17:00.348848 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.348085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.0.98.61:44419)
0620 00:17:00.348179 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.348181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.348244 (+ 63us) server_negotiation.cc:407] Connection header received
0620 00:17:00.348302 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.348303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.348313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.348364 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.348773 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44798: BlockingRecv error: recv got EOF from 127.0.0.1:44798 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 00:17:00.607221 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:00.606464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.0.98.61:44419)
0620 00:17:00.606557 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:00.606560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:00.606571 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:17:00.606603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:00.606604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:00.606620 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:00.606696 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:00.607123 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44800: BlockingRecv error: recv got EOF from 127.0.0.1:44800 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 00:17:00.965960 22879 catalog_manager.cc:1259] Loaded cluster ID: a76e0569cde040bea04d59b217cdc1b2
I20260620 00:17:00.966045 22879 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: loading cluster ID for follower catalog manager: success
I20260620 00:17:00.966930 22879 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: acquiring CA information for follower catalog manager: success
I20260620 00:17:00.967270 22879 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d5285f652d04233b017179efe4d40b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:17:01.017326 22899 catalog_manager.cc:1259] Loaded cluster ID: a76e0569cde040bea04d59b217cdc1b2
I20260620 00:17:01.017409 22899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: loading cluster ID for follower catalog manager: success
I20260620 00:17:01.018389 22899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: acquiring CA information for follower catalog manager: success
I20260620 00:17:01.018775 22899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aeab81a4d75948fc8ee703d59799fe0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:17:01.123507 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:01.120675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.61:44419)
0620 00:17:01.120851 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:01.120853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:01.120867 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:17:01.120981 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:01.120982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:01.121038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:01.121116 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:01.121348 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.121890 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.122679 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.122888 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.122942 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:01.123092 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:01.123203 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:01.123270 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:01.123365 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:17:01.123384 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260620 00:17:01.129199 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:01.126288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.98.60:41721)
0620 00:17:01.126436 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:01.126437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:01.126446 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:01.126487 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:01.126488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:01.126528 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:01.126613 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:01.126824 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.127325 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.128232 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.128436 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.128507 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:01.128817 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:01.128934 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:01.129004 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:01.129091 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 00:17:01.129122 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260620 00:17:01.135128 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:01.131788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.0.98.61:44419)
0620 00:17:01.131835 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:01.131837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:01.132245 (+ 408us) server_negotiation.cc:407] Connection header received
0620 00:17:01.132339 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:01.132340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:01.132355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:01.132411 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:01.132534 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.133286 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134138 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.134435 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134476 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:01.134716 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:01.134748 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:01.134792 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:01.135002 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 00:17:01.135036 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:17:01.135324 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:01.131556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.98.62:46071)
0620 00:17:01.131759 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:01.131761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:01.131790 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:17:01.131905 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:01.131905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:01.131945 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:01.132093 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:01.132244 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.132910 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134044 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.134356 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134427 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:01.134637 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:01.134883 (+ 246us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:01.134955 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:01.135181 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 00:17:01.135225 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":66,"threads_started":1}
I20260620 00:17:01.135425 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:01.131684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.0.98.60:41721)
0620 00:17:01.131748 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:01.131750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:01.132102 (+ 352us) server_negotiation.cc:407] Connection header received
0620 00:17:01.132171 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:01.132172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:01.132183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:01.132239 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:01.132369 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.133219 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134251 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:01.134504 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:01.134582 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:01.135169 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:01.135209 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:01.135258 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:01.135317 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:17:01.135340 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:17:01.137986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22619
I20260620 00:17:01.144483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22694
I20260620 00:17:01.150388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 22769
2026-06-20T00:17:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2903 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:01 dist-test-slave-28bv krb5kdc[22988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:01 dist-test-slave-28bv krb5kdc[22988](info): set up 2 sockets
Jun 20 00:17:01 dist-test-slave-28bv krb5kdc[22988](info): commencing operation
krb5kdc: starting...
W20260620 00:17:02.245249 392 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/test-user.keytab.
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:17:02.385726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:02.477914 23001 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:17:02.478188 23001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:02.478210 23001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:02.478235 23001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:02.479568 23001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:02.479635 23001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:02.479652 23001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:02.479665 23001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:02.479678 23001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:02.481412 23001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:02.481747 23001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:02.482031 23001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:02.488659 23009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:02.488653 23006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:02.488653 23007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:02.489568 23001 server_base.cc:1034] running on GCE node
I20260620 00:17:02.489923 23001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:02.490263 23001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:02.491446 23001 hybrid_clock.cc:648] HybridClock initialized: now 1781914622491400 us; error 69 us; skew 500 ppm
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:02.500021 23001 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:17:02.500705 23001 webserver.cc:460] Webserver started at http://127.0.98.62:38025/ using document root <none> and password file <none>
I20260620 00:17:02.500973 23001 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:02.501025 23001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:02.501173 23001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:02.502414 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data/instance:
uuid: "df6842b821df431ab9ab26f1488444cc"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "2f1f6fc57133005219de0d95c97f975a"
server_key_iv: "ce12a6fdfc438f0103710a4b3def933b"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.502820 23001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance:
uuid: "df6842b821df431ab9ab26f1488444cc"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "2f1f6fc57133005219de0d95c97f975a"
server_key_iv: "ce12a6fdfc438f0103710a4b3def933b"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.504423 23001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:17:02.505900 23016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:02.506315 23001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:02.506441 23001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal
uuid: "df6842b821df431ab9ab26f1488444cc"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "2f1f6fc57133005219de0d95c97f975a"
server_key_iv: "ce12a6fdfc438f0103710a4b3def933b"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.506518 23001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:02.523447 23001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:02.526083 23001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:02.526216 23001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:02.530680 23001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46189
I20260620 00:17:02.530947 23068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46189 every 8 connection(s)
I20260620 00:17:02.531230 23001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:17:02.531908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 23001
I20260620 00:17:02.531988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-0/wal/instance
I20260620 00:17:02.532251 392 external_mini_cluster.cc:1428] Setting key 053545ef5b192a7833f427bfe355bd70
I20260620 00:17:02.532737 23069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:02.535037 23069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:02.550549 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.535974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46189 (local address 127.0.0.1:34476)
0620 00:17:02.536168 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.536224 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.537014 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.537321 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.537323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.537645 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.537808 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.537813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.538867 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.538868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.539634 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.539636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.540193 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.540195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.540261 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.540881 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.540890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.544821 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.548070 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.548081 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.548084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.548630 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.549311 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.549312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.549312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.549421 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.549874 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.549879 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.549950 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.550056 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.550334 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":104,"thread_start_us":113,"threads_started":1}
I20260620 00:17:02.550546 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.536000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.0.98.62:46189)
0620 00:17:02.536250 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.536255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.536315 (+ 60us) server_negotiation.cc:407] Connection header received
0620 00:17:02.537198 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.537200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.537218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.537317 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.537915 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.538759 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.539741 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.540065 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.540212 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.544968 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.544979 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.544979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.544989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.547824 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.549072 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.549077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.549078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.549224 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.549514 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.549515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.549516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.549792 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.549892 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.550095 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.550263 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260620 00:17:02.551647 23069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.552150 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.552012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42388)
0620 00:17:02.552070 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.552124 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 00:17:02.552294 23020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
I20260620 00:17:02.552786 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.552682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42394)
0620 00:17:02.552734 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.552767 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:17:02.552888 23069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:17:02.565248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38765
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:17:02.602242 23069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } attempt: 1
I20260620 00:17:02.602880 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.602722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42396)
0620 00:17:02.602793 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.602846 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:17:02.603330 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.603236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42398)
0620 00:17:02.603283 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.603312 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:17:02.603490 23069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
W20260620 00:17:02.655220 23076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:17:02.655498 23076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:02.655529 23076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:02.655599 23076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:02.656891 23076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:02.656955 23076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:02.656977 23076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:02.656999 23076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:02.657029 23076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:02.658581 23076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:02.658870 23076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:02.659109 23076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:02.664353 23082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:02.664353 23081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:02.664373 23084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:02.664718 23076 server_base.cc:1034] running on GCE node
I20260620 00:17:02.664988 23076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:02.665262 23076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:02.666426 23076 hybrid_clock.cc:648] HybridClock initialized: now 1781914622666365 us; error 91 us; skew 500 ppm
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:02.669305 23076 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:17:02.669870 23076 webserver.cc:460] Webserver started at http://127.0.98.61:32899/ using document root <none> and password file <none>
I20260620 00:17:02.670117 23076 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:02.670192 23076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:02.670361 23076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:02.671449 23076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data/instance:
uuid: "29402a0d11c4451f85be0a3078d3ab76"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "04b92e3ec6a63b53c50408a471ca37a3"
server_key_iv: "24f21fc31ed897c64ff3d6a8e0cc1a96"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.671720 23069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } attempt: 2
I20260620 00:17:02.671865 23076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal/instance:
uuid: "29402a0d11c4451f85be0a3078d3ab76"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "04b92e3ec6a63b53c50408a471ca37a3"
server_key_iv: "24f21fc31ed897c64ff3d6a8e0cc1a96"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.672124 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.671990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42410)
0620 00:17:02.672051 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.672083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:17:02.672464 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.672371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42424)
0620 00:17:02.672413 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.672438 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 00:17:02.672561 23069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38765: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38765: connect: Connection refused (error 111)
I20260620 00:17:02.673374 23076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:17:02.674257 23091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:02.674422 23076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:17:02.674530 23076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal
uuid: "29402a0d11c4451f85be0a3078d3ab76"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "04b92e3ec6a63b53c50408a471ca37a3"
server_key_iv: "24f21fc31ed897c64ff3d6a8e0cc1a96"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.674605 23076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:02.689052 23076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:02.691315 23076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:02.691500 23076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:02.696184 23076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38765
I20260620 00:17:02.696262 23143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38765 every 8 connection(s)
I20260620 00:17:02.696705 23076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:17:02.697798 23144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:02.699445 23144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.700894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 23076
I20260620 00:17:02.700994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-1/wal/instance
I20260620 00:17:02.701191 392 external_mini_cluster.cc:1428] Setting key 2e930414ec8c1179ef2e228e5be01d89
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 00:17:02.711336 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.700351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34492 (local address 127.0.98.62:46189)
0620 00:17:02.700488 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.700491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.701174 (+ 683us) server_negotiation.cc:407] Connection header received
0620 00:17:02.701604 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.701606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.701627 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.701705 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.702686 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.703521 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.704813 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.705181 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.705265 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.707815 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.707816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.707817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.707826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.709589 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.710168 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.710169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.710170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.710253 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.710539 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.710540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.710540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.710716 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.710783 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.711091 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.711227 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 00:17:02.711360 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.700303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46189 (local address 127.0.0.1:34492)
0620 00:17:02.700580 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.700638 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.701403 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.701838 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.701843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.702248 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.702548 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.702555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.703649 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.703650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.704694 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.704696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.705307 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.705308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.705423 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.706020 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.706028 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.707713 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.709753 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.709762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.709764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.710061 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.710327 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.710328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.710328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.710448 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.710808 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.710814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.710944 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.711035 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.711179 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 00:17:02.712106 23144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:17:02.721189 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.712513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.0.98.61:38765)
0620 00:17:02.712707 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.712712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.712735 (+ 23us) server_negotiation.cc:407] Connection header received
0620 00:17:02.712759 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.712762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.712778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.712860 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.713173 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.714122 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.715194 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.715514 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.715574 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.717810 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.717821 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.717821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.717830 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.719801 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.720209 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.720213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.720214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.720283 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.720485 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.720486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.720486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.720738 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.720841 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.721000 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.721081 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":118,"threads_started":1}
I20260620 00:17:02.721268 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.712434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42434)
0620 00:17:02.712489 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.712502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.712567 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.712905 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.712906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.713040 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.713094 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.713096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.714216 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.714218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.715094 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.715097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.715603 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.715604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.715693 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.716212 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.716218 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.717716 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.719933 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.719934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.719934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.720114 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.720357 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.720358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.720358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.720418 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.720836 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.720837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.720879 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.720950 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.721133 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":48}
I20260620 00:17:02.722091 23144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.722493 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.722361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53774)
0620 00:17:02.722421 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.722471 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 00:17:02.722680 23093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
I20260620 00:17:02.723248 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.723152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53776)
0620 00:17:02.723195 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.723226 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 00:17:02.723368 23144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:17:02.732115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUZKDhK/build/release/bin/kudu
/tmp/dist-test-taskUZKDhK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46361
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:17:02.763746 23069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } attempt: 3
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:17:02.772169 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.764304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.0.98.61:38765)
0620 00:17:02.764378 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.764380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.764390 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:02.764442 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.764443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.764454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.764602 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.764940 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.765469 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.766440 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.766664 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.766745 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.768826 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.768828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.768828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.768844 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.770341 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.771013 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.771014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.771014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.771083 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.771517 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.771519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.771519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.771707 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.771785 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.772014 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.772101 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:17:02.772169 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.764225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42444)
0620 00:17:02.764285 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.764305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.764374 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.764617 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.764618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.764785 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.764851 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.764853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.765574 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.765576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.766328 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.766331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.766742 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.766743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.766824 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.767387 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.767392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.768731 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.770630 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.770632 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.770632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.770922 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.771274 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.771275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.771275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.771366 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.771802 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.771804 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.771863 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.771942 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.772056 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:17:02.772580 23144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attempt: 1
I20260620 00:17:02.772702 23069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.772991 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.772855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53790)
0620 00:17:02.772914 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.772953 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:17:02.773126 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.772909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53802)
0620 00:17:02.773077 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.773105 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 00:17:02.773411 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.773309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53812)
0620 00:17:02.773364 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.773391 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:17:02.773545 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.773336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53826)
0620 00:17:02.773499 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.773523 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260620 00:17:02.773555 23144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
W20260620 00:17:02.773613 23069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
I20260620 00:17:02.804944 23069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attempt: 1
I20260620 00:17:02.805495 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.805339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53840)
0620 00:17:02.805411 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.805457 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:17:02.805960 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.805854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53854)
0620 00:17:02.805903 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.805935 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:17:02.806115 23069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
W20260620 00:17:02.821661 23151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:17:02.821906 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:02.821923 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:02.821947 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:02.823094 23151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:02.823136 23151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:02.823158 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:02.823172 23151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:02.823184 23151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:02.824761 23151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46189,127.0.98.61:38765,127.0.98.60:46361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:02.824987 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:02.825173 23151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:02.830590 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:02.830595 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:02.830794 23151 server_base.cc:1034] running on GCE node
W20260620 00:17:02.830663 23159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:02.831188 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:02.831553 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:02.832669 23151 hybrid_clock.cc:648] HybridClock initialized: now 1781914622832652 us; error 32 us; skew 500 ppm
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:02.835354 23151 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:17:02.835810 23151 webserver.cc:460] Webserver started at http://127.0.98.60:42397/ using document root <none> and password file <none>
I20260620 00:17:02.836097 23151 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:02.836176 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:02.836328 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:02.837939 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data/instance:
uuid: "eccc2ff04d7945eab640b06f11cb7d93"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "831ddadf87a5fc6eb6d99a0feee4e5a7"
server_key_iv: "4503952b9d3bdedaf22868227fe37783"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.838409 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal/instance:
uuid: "eccc2ff04d7945eab640b06f11cb7d93"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "831ddadf87a5fc6eb6d99a0feee4e5a7"
server_key_iv: "4503952b9d3bdedaf22868227fe37783"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.840160 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:17:02.841123 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:02.841392 23151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:17:02.841569 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal
uuid: "eccc2ff04d7945eab640b06f11cb7d93"
format_stamp: "Formatted at 2026-06-20 00:17:02 on dist-test-slave-28bv"
server_key: "831ddadf87a5fc6eb6d99a0feee4e5a7"
server_key_iv: "4503952b9d3bdedaf22868227fe37783"
server_key_version: "encryptionkey@0"
I20260620 00:17:02.841658 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:02.841823 23144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attempt: 2
I20260620 00:17:02.842274 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.842113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53862)
0620 00:17:02.842186 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.842233 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:17:02.842707 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.842604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53864)
0620 00:17:02.842654 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.842685 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:17:02.842854 23144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46361: connect: Connection refused (error 111)
I20260620 00:17:02.856853 23151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:02.859268 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:02.859444 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:02.863972 23151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46361
I20260620 00:17:02.864018 23218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46361 every 8 connection(s)
I20260620 00:17:02.864429 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:17:02.865464 23219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:02.867158 23219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.867640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUZKDhK/build/release/bin/kudu as pid 23151
I20260620 00:17:02.867744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUZKDhK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914322045185-392-0/minicluster-data/master-2/wal/instance
I20260620 00:17:02.867990 392 external_mini_cluster.cc:1428] Setting key a937f0f5ad8fd6449cf3b025c4cecf8d
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:17:02.877919 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.867986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.0.98.62:46189)
0620 00:17:02.868035 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.868037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.868376 (+ 339us) server_negotiation.cc:407] Connection header received
0620 00:17:02.869119 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.869121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.869132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.869205 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.869991 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.870562 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.871685 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.871958 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.872011 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.874702 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.874703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.874704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.874713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.876238 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.876805 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.876807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.876808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.876875 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.877211 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.877211 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.877212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.877335 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.877384 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.877699 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.877784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:17:02.878118 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.867986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46189 (local address 127.0.0.1:34500)
0620 00:17:02.868176 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.868230 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.868977 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.869294 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.869298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.869632 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.869863 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.869871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.870729 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.870730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.871540 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.871543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.872055 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.872056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.872191 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.872874 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.872883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.874557 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.876364 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.876372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.876374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.876713 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.876965 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.876965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.876966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.877139 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.877382 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.877389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.877553 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.877676 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.877871 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260620 00:17:02.878662 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.868461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.0.98.60:46361)
0620 00:17:02.868641 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.868646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.868683 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:17:02.868978 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.868987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.869023 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.869113 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.869940 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.870936 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.871866 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.872293 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.872373 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.875211 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.875217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.875217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.875223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.877061 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.877487 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.877490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.877490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.877546 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.877835 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.877836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.877836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.878057 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.878219 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.878475 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.878558 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":121,"threads_started":1}
I20260620 00:17:02.878690 23219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.881254 23069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attempt: 2
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:17:02.886719 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.878929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38765 (local address 127.0.0.1:42458)
0620 00:17:02.879149 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.879158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.879229 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.879350 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.879350 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.879491 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.879525 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.879527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.880197 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.880198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.880783 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.880785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.881173 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.881174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.881258 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.881707 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.881713 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.883032 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.885320 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.885321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.885321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.885619 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.885925 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.885925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.885926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.886011 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.886359 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.886360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.886407 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.886484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.886595 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 00:17:02.886739 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.878997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.98.61:38765)
0620 00:17:02.879055 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.879057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.879216 (+ 159us) server_negotiation.cc:407] Connection header received
0620 00:17:02.879291 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.879292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.879304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.879349 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.879643 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.880110 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.880873 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.881084 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.881168 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.883114 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.883116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.883116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.883124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.884988 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.885734 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.885735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.885736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.885804 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.886098 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.886099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.886100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.886254 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.886306 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.886535 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.886636 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:17:02.887193 23219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:17:02.889525 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.879667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.0.98.62:46189)
0620 00:17:02.879706 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.879708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.879715 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:17:02.879737 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.879738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.879746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.879819 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.880304 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.880829 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.882026 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.882664 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.882743 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.885393 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.885395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.885395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.885403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.887962 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.888464 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.888466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.888466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.888575 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.888805 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.888806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.888806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.888963 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.889172 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.889215 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.889367 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:02.890849 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.881592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.0.98.60:46361)
0620 00:17:02.881630 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.881632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.881654 (+ 22us) server_negotiation.cc:407] Connection header received
0620 00:17:02.881743 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.881744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.881755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.881795 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.882437 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.883123 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.884338 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.884751 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.884836 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.887033 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.887035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.887036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.887044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.888938 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.889472 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.889473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.889473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.889534 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.890215 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.890216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.890217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.890370 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.890467 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.890666 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.890747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:17:02.890851 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.881554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53878)
0620 00:17:02.881610 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.881620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.881677 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.881813 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.881814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.881955 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.882028 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.882029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.883220 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.883221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.884216 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.884220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.884845 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.884846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.884914 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.885328 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.885334 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.886916 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.889077 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.889078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.889078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.889374 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.890030 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.890031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.890032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.890111 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.890464 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.890465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.890515 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.890634 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.890747 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:17:02.893556 23069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: Bootstrap starting.
I20260620 00:17:02.894222 23069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:02.894474 23069 log.cc:824] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:02.895387 23069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: No bootstrap required, opened a new log
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:17:02.896562 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.887493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53890)
0620 00:17:02.887531 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.887542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.887588 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.887903 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.887903 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.888288 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.888324 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.888326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.889232 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.889233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.889995 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.889998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.890999 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.891000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.891048 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.891467 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.891472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.893008 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.895082 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.895083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.895084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.895319 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.895560 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.895561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.895561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.895616 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.896121 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.896122 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.896171 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.896257 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.896367 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 00:17:02.896968 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.887528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.0.98.60:46361)
0620 00:17:02.887737 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.887739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.887748 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:02.887768 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.887769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.887780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.887854 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.888406 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.889155 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.890075 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.890879 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.890950 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.893090 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.893092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.893092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.893099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.894980 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.895410 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.895411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.895411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.895472 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.895694 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.895696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.895696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.895905 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.896114 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.896304 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.896436 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":179,"thread_start_us":121,"threads_started":1}
I20260620 00:17:02.897951 23069 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.898458 23069 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6842b821df431ab9ab26f1488444cc, State: Initialized, Role: FOLLOWER
I20260620 00:17:02.898561 23069 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.898937 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.898993 23069 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:02.899049 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.899261 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: Bootstrap starting.
I20260620 00:17:02.899399 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.890227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.0.98.61:38765)
0620 00:17:02.890272 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.890274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.890281 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:17:02.890306 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.890306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.890318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.890383 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.891043 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.892079 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.892903 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.893238 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.893328 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.896493 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.896494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.896494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.896503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.898048 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.898417 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.898418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.898419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.898482 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.898699 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.898700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.898700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.898835 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.899003 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.899203 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.899279 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:17:02.900048 23219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:02.900398 23219 log.cc:824] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:02.900833 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:17:02.901202 23219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: No bootstrap required, opened a new log
W20260620 00:17:02.901575 23238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:17:02.901641 23238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:17:02.901719 23043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "df6842b821df431ab9ab26f1488444cc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34516
I20260620 00:17:02.901823 23043 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:17:02.901863 23043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:02.903170 23043 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.903419 23043 leader_election.cc:288] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [CANDIDATE]: Term 1 election: Requested vote from peers 29402a0d11c4451f85be0a3078d3ab76 (127.0.98.61:38765), eccc2ff04d7945eab640b06f11cb7d93 (127.0.98.60:46361)
I20260620 00:17:02.903461 23219 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.903705 23219 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccc2ff04d7945eab640b06f11cb7d93, State: Initialized, Role: FOLLOWER
I20260620 00:17:02.903792 23219 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.904093 23219 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:02.902045 23001 master_runner.cc:186] Error getting master registration for 127.0.98.61:38765: OK, instance_id { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" instance_seqno: 1781914622694609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:17:02.904371 23001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:17:02.904491 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df6842b821df431ab9ab26f1488444cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "eccc2ff04d7945eab640b06f11cb7d93"
I20260620 00:17:02.904683 23193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:02.905989 23193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6842b821df431ab9ab26f1488444cc in term 1.
I20260620 00:17:02.906452 23018 leader_election.cc:302] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6842b821df431ab9ab26f1488444cc, eccc2ff04d7945eab640b06f11cb7d93; no voters:
I20260620 00:17:02.906589 23226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:02.906755 23226 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [term 1 LEADER]: Becoming Leader. State: Replica: df6842b821df431ab9ab26f1488444cc, State: Running, Role: LEADER
I20260620 00:17:02.906848 23226 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
W20260620 00:17:02.906890 23255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:17:02.907034 23255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:17:02.908345 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6842b821df431ab9ab26f1488444cc. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.908457 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:02.908902 23151 master_runner.cc:186] Error getting master registration for 127.0.98.61:38765: OK, instance_id { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" instance_seqno: 1781914622694609 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:17:02.911295 23151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:17:02.913017 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.913127 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.917526 23259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:17:02.917794 23259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:17:02.923996 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.909118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.98.62:46189)
0620 00:17:02.909155 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.909158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.912935 (+ 3777us) server_negotiation.cc:407] Connection header received
0620 00:17:02.913306 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.913308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.913318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.913350 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.914357 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.915181 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.916289 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.916628 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.916675 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.919946 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.919948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.919948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.919957 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.921935 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.922393 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.922395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.922395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.922457 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.922708 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.922708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.922709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.922868 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.922985 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.923796 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.923886 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:17:02.924665 23193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [term 1 FOLLOWER]: Refusing update from remote peer df6842b821df431ab9ab26f1488444cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:02.925001 23257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:02.925747 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.908830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.0.98.60:46361)
0620 00:17:02.908878 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.908881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.912774 (+ 3893us) server_negotiation.cc:407] Connection header received
0620 00:17:02.912847 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.912848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.912875 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.912930 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.915367 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.916303 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.919313 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.919676 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.919731 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.921776 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.921778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.921778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.921786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.923524 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.924603 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.924604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.924604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.924678 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.924971 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.924972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.924972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.925110 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.925224 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.925539 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.925640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:17:02.926579 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6842b821df431ab9ab26f1488444cc. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.926702 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.928004 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.928103 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:02.928344 23263 mvcc.cc:205] Tried to move back new op lower bound from 7298722295493210112 to 7298722295427174400. Current Snapshot: MvccSnapshot[applied={T|T < 7298722295493210112}]
I20260620 00:17:02.928707 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.928793 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.929236 23259 catalog_manager.cc:1347] Generated new cluster ID: c020a8c98fc74ce0ba3dea984008b428
I20260620 00:17:02.929342 23259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:17:02.934015 23144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } attempt: 3
Jun 20 00:17:02 dist-test-slave-28bv krb5kdc[22988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:17:02.940963 23259 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:02.941558 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.909334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.98.61:38765)
0620 00:17:02.909383 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.909385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.914259 (+ 4874us) server_negotiation.cc:407] Connection header received
0620 00:17:02.914342 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.914343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.914355 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.914401 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.917358 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.918158 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.920110 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.920445 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.920491 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.936481 (+ 15990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.936483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.936483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.936493 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.939305 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.939765 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.939767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.939768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.939840 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.940061 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.940062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.940062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.940498 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.940667 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.941378 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.941458 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:17:02.943889 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.934399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46361 (local address 127.0.0.1:53914)
0620 00:17:02.934554 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:02.934566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:02.934609 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:02.934951 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:02.934952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:02.935172 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:02.935222 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.935223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.936276 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.936277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.937170 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:02.937174 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.937755 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.937757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:02.937820 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.938254 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:02.938260 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:02.940349 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:02.942715 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.942716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.942717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.942950 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.943184 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:02.943185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:02.943185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:02.943247 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:02.943564 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:02.943564 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:02.943609 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:17:02.943671 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:17:02.943782 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260620 00:17:02.944384 23259 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:17:02.945327 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.934767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.0.98.60:46361)
0620 00:17:02.934807 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.934809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.934817 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:17:02.934843 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.934844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.934853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.934899 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.935303 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.936167 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.937312 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.937654 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:02.937758 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:02.940439 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:02.940441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:02.940441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:02.940450 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:02.942567 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.943035 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.943036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.943036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.943100 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:02.943314 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:02.943315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:02.943315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:02.943479 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:02.943537 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:02.944285 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 00:17:02.944371 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:17:02.946864 23144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: Bootstrap starting.
I20260620 00:17:02.947607 23144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:02.947924 23144 log.cc:824] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:02.953368 23144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: No bootstrap required, opened a new log
I20260620 00:17:02.955452 23144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.955618 23144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29402a0d11c4451f85be0a3078d3ab76, State: Initialized, Role: FOLLOWER
I20260620 00:17:02.955711 23144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } }
I20260620 00:17:02.956056 23144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:02.956599 23118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df6842b821df431ab9ab26f1488444cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29402a0d11c4451f85be0a3078d3ab76"
I20260620 00:17:02.956704 23118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:02.957578 23118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6842b821df431ab9ab26f1488444cc in term 1.
I20260620 00:17:02.957947 23117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [term 1 FOLLOWER]: Refusing update from remote peer df6842b821df431ab9ab26f1488444cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:02.958325 23257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:02.960793 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6842b821df431ab9ab26f1488444cc. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.960897 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.960961 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.961011 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.961439 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.961558 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:02.961961 23276 mvcc.cc:205] Tried to move back new op lower bound from 7298722295553683456 to 7298722295427174400. Current Snapshot: MvccSnapshot[applied={T|T < 7298722295493210112}]
I20260620 00:17:02.964804 23259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6842b821df431ab9ab26f1488444cc: Generated new TSK 0
I20260620 00:17:02.965036 23259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:17:02.960301 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6842b821df431ab9ab26f1488444cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6842b821df431ab9ab26f1488444cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46189 } } peers { permanent_uuid: "29402a0d11c4451f85be0a3078d3ab76" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38765 } } peers { permanent_uuid: "eccc2ff04d7945eab640b06f11cb7d93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46361 } } }
I20260620 00:17:02.965476 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:17:02.970510 23284 catalog_manager.cc:1259] Loaded cluster ID: c020a8c98fc74ce0ba3dea984008b428
I20260620 00:17:02.970579 23284 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: loading cluster ID for follower catalog manager: success
I20260620 00:17:02.971995 23284 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: acquiring CA information for follower catalog manager: success
I20260620 00:17:02.972415 23284 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 29402a0d11c4451f85be0a3078d3ab76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 00:17:02.990837 23073 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:02.989049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.0.98.62:46189)
0620 00:17:02.989101 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.989103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.990127 (+ 1024us) server_negotiation.cc:407] Connection header received
0620 00:17:02.990238 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.990239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.990251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.990284 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.990763 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34532: BlockingRecv error: recv got EOF from 127.0.0.1:34532 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 00:17:02.991917 23148 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:02.989428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.98.61:38765)
0620 00:17:02.989488 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.989490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.991213 (+ 1723us) server_negotiation.cc:407] Connection header received
0620 00:17:02.991448 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.991449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.991461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.991491 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.991854 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42484: BlockingRecv error: recv got EOF from 127.0.0.1:42484 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 00:17:02.992702 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:02.989838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.0.98.60:46361)
0620 00:17:02.989880 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.989882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.991395 (+ 1513us) server_negotiation.cc:407] Connection header received
0620 00:17:02.992246 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.992248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.992266 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.992297 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:02.992628 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53928: BlockingRecv error: recv got EOF from 127.0.0.1:53928 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:03.001091 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:02.997270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.0.98.61:38765)
0620 00:17:02.997317 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:02.997318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:02.997516 (+ 198us) server_negotiation.cc:407] Connection header received
0620 00:17:02.997623 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:02.997624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:02.997670 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:02.997730 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:02.997949 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:02.998754 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.000004 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.000316 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.000362 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.000666 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.000814 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.000869 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.000984 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.001004 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:17:03.005838 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.004895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.98.60:46361)
0620 00:17:03.004936 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.004938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.005202 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:17:03.005313 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.005314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.005333 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.005374 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.005777 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53944: BlockingRecv error: recv got EOF from 127.0.0.1:53944 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:17:03.011822 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.008368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.61:38765)
0620 00:17:03.008417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.008419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.008668 (+ 249us) server_negotiation.cc:407] Connection header received
0620 00:17:03.008773 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.008774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.008784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.008812 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.008983 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.009838 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.011015 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.011315 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.011355 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.011539 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.011572 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.011598 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.011717 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.011739 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:17:03.015820 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.014800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.60:46361)
0620 00:17:03.014860 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.014862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.015178 (+ 316us) server_negotiation.cc:407] Connection header received
0620 00:17:03.015292 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.015293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.015305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.015339 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.015749 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53950: BlockingRecv error: recv got EOF from 127.0.0.1:53950 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:17:03.022806 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.019376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.98.61:38765)
0620 00:17:03.019419 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.019421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.019697 (+ 276us) server_negotiation.cc:407] Connection header received
0620 00:17:03.019797 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.019798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.019808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.019837 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.020015 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.020830 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.021982 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.022286 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.022327 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.022523 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.022567 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.022594 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.022697 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.022718 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:17:03.026278 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.025452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.0.98.60:46361)
0620 00:17:03.025494 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.025496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.025717 (+ 221us) server_negotiation.cc:407] Connection header received
0620 00:17:03.025813 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.025814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.025825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.025854 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.026202 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53958: BlockingRecv error: recv got EOF from 127.0.0.1:53958 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:17:03.035226 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.031771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.0.98.61:38765)
0620 00:17:03.031817 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.031819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.032115 (+ 296us) server_negotiation.cc:407] Connection header received
0620 00:17:03.032219 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.032220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.032231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.032260 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.032442 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.033261 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.034410 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.034712 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.034752 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.034946 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.034989 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.035017 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.035121 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.035141 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:17:03.038693 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.037878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.0.98.60:46361)
0620 00:17:03.037924 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.037926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.038133 (+ 207us) server_negotiation.cc:407] Connection header received
0620 00:17:03.038231 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.038232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.038243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.038273 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.038626 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53966: BlockingRecv error: recv got EOF from 127.0.0.1:53966 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:17:03.051731 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.048199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.0.98.61:38765)
0620 00:17:03.048245 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.048248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.048579 (+ 331us) server_negotiation.cc:407] Connection header received
0620 00:17:03.048694 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.048695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.048706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.048735 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.048928 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.049725 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.050888 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.051199 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.051240 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.051439 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.051473 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.051500 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.051603 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.051625 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:17:03.055317 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.054494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.0.98.60:46361)
0620 00:17:03.054536 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.054539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.054741 (+ 202us) server_negotiation.cc:407] Connection header received
0620 00:17:03.054836 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.054837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.054850 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.054879 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.055252 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53980: BlockingRecv error: recv got EOF from 127.0.0.1:53980 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:03.076635 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.072881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.0.98.61:38765)
0620 00:17:03.072929 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.072932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.073261 (+ 329us) server_negotiation.cc:407] Connection header received
0620 00:17:03.073376 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.073377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.073389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.073422 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.073626 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.074515 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.075688 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.075968 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.076014 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.076253 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.076288 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.076318 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.076465 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.076487 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:17:03.080423 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.079491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.60:46361)
0620 00:17:03.079536 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.079538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.079829 (+ 291us) server_negotiation.cc:407] Connection header received
0620 00:17:03.079930 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.079931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.079942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.079972 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.080365 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53990: BlockingRecv error: recv got EOF from 127.0.0.1:53990 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:03.117244 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.114008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.0.98.61:38765)
0620 00:17:03.114061 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.114064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.114167 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:17:03.114213 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.114215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.114227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.114323 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.114465 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.115194 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.116340 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.116698 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.116777 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.116953 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.116986 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.117048 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.117137 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.117159 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 00:17:03.119604 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.119081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.0.98.60:46361)
0620 00:17:03.119120 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.119122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.119131 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:03.119155 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.119155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.119165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.119219 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.119542 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54002: BlockingRecv error: recv got EOF from 127.0.0.1:54002 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:17:03.187942 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.185234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.0.98.61:38765)
0620 00:17:03.185311 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.185313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.185415 (+ 102us) server_negotiation.cc:407] Connection header received
0620 00:17:03.185636 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.185638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.185648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.185708 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.185859 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.186358 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.187201 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.187387 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.187453 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.187676 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.187710 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.187753 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.187812 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.187829 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 00:17:03.190312 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.189804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.98.60:46361)
0620 00:17:03.189858 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.189860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.189899 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:17:03.189953 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.189954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.189963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.190004 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.190266 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54010: BlockingRecv error: recv got EOF from 127.0.0.1:54010 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:17:03.322561 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.319825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.0.98.61:38765)
0620 00:17:03.319936 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.319938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.319957 (+ 19us) server_negotiation.cc:407] Connection header received
0620 00:17:03.319985 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.319985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.319995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.320061 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.320241 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.320755 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.321779 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.321991 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.322057 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.322231 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.322265 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.322329 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.322462 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.322493 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 00:17:03.325429 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.324671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.0.98.60:46361)
0620 00:17:03.324723 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.324725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.324808 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:17:03.324912 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.324913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.324929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.325000 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.325353 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54016: BlockingRecv error: recv got EOF from 127.0.0.1:54016 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:17:03.585558 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:03.583078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.98.61:38765)
0620 00:17:03.583146 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.583148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.583160 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:17:03.583227 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.583228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.583237 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.583301 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:03.583422 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.583940 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.584866 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:03.585073 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:03.585139 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:03.585308 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:03.585336 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:03.585376 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:03.585433 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:17:03.585450 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 00:17:03.588358 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:17:03.587760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.98.60:46361)
0620 00:17:03.587832 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:03.587835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:03.587843 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:17:03.587870 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:03.587871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:03.587882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:03.587933 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:03.588267 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54030: BlockingRecv error: recv got EOF from 127.0.0.1:54030 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:17:03.907588 23255 catalog_manager.cc:1259] Loaded cluster ID: c020a8c98fc74ce0ba3dea984008b428
I20260620 00:17:03.907711 23255 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: loading cluster ID for follower catalog manager: success
I20260620 00:17:03.908569 23255 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: acquiring CA information for follower catalog manager: success
I20260620 00:17:03.908912 23255 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eccc2ff04d7945eab640b06f11cb7d93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:17:04.104707 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:04.101737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.0.98.61:38765)
0620 00:17:04.101909 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:04.101911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:04.102016 (+ 105us) server_negotiation.cc:407] Connection header received
0620 00:17:04.102126 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:04.102127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:04.102142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:04.102224 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:04.102365 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.102924 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.103824 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.104097 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.104197 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:04.104360 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:04.104396 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:04.104468 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:04.104531 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 00:17:04.104581 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260620 00:17:04.109228 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:04.106641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.0.98.60:46361)
0620 00:17:04.106765 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:04.106767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:04.106775 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:17:04.106799 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:04.106800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:04.106849 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:04.106921 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:04.107019 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.107507 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.108337 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.108626 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.108690 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:04.108832 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:04.108967 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:04.109029 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:04.109136 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 00:17:04.109156 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260620 00:17:04.114737 23409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:04.111618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.98.60:46361)
0620 00:17:04.111671 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:04.111674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:04.111837 (+ 163us) server_negotiation.cc:407] Connection header received
0620 00:17:04.111904 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:04.111905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:04.111917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:04.111973 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:04.112074 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.112753 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.113789 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.114106 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.114195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:04.114389 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:04.114431 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:04.114483 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:04.114610 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 00:17:04.114642 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:17:04.114944 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:04.111737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.98.61:38765)
0620 00:17:04.111784 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:04.111786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:04.111924 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:17:04.112022 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:04.112024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:04.112038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:04.112092 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:04.112279 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.113113 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.114071 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.114377 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.114449 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:04.114630 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:04.114669 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:04.114700 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:04.114828 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 00:17:04.114860 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:17:04.115407 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:04.111504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.0.98.62:46189)
0620 00:17:04.112035 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:04.112037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:04.112046 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:04.112092 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:04.112092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:04.112148 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:04.112243 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:04.112391 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.113189 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.114175 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:04.114529 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:04.114591 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:04.114829 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:04.114966 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:04.115043 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:04.115258 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 00:17:04.115292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":62,"threads_started":1}
I20260620 00:17:04.117697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 23001
I20260620 00:17:04.123605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 23076
I20260620 00:17:04.128684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUZKDhK/build/release/bin/kudu with pid 23151
2026-06-20T00:17:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2978 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5882 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302083 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS